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PREFACE 



How can we best foster accountability in vocational education? What is the nature of 
existing accountability relationships? Concern about the productivity of the U.S. workforce 
and the competitiveness of the U.S. economy heightens the importance of these questions. 
Recent reports from the U.S. Department of Labor and the National Educational Goals Panel 
focused public attention on the performance of the vocational training system. Legislators at 
state and federal levels are formulating policies aimed at improving vocational education 
through increased accouTxtability. These policies will be better informed if they are based on 
thorough knowledge of existing accountability systems. 

Yet we know very little about accoimtability in vocational programs. At the local level, 
schools and programs have long-standing relationships with constituents through which 
their performance is measured and judged. These local accountability systems, although 
often informal, have many of the same elements that are being incorporated into formal 
regulations. However, there is no clear picture of the functioning of such local accountability 
systems. 

This study was designed to analyze and describe local accountability systems in 
vocational education. Such a description can serve as a basis for further study of vocational 
programs, for developing criteria for evaluating accountability systems, and f^r monitoring 
the impact of policy changes. As such, this Note should be of interest to state and local 
education administrators, state and federal policymakers, and policy researchers interested 
in vocational education. 

The present Note describes a model for local accoimtability in vocational education and 
explores rhortcomings in accountability systems. A subsequent report will consider the 
demands that federal legislation places on states and localities and will propose policy 
alternatives for responding to these demands that are sensitive to local accountability 
concerns. RAND research on vocational education is conducted within the National Center 
for Research in Vocational Education, sponsored by the Office of Vocational and Adult 
Education, U.S. Department of Education. 
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SUMMARY 



Accountability refers to the practice of holding educational systems responsible for the 
quality of their products. Most formal policy initiatives in this area focus on highly 
aggregated accountability at the state or federal level, but it is also possible to define 
accotmtability systems in local terms. The Carl D. Perkins Vocational and Applied 
Technology Education Act of 1990 attempts to promote both state and local accoiintability, 
placing with states the final responsibility for establishing measures and standards and for 
idtimate program supenision after placing initial responsibility at the local level for program 
evaluation and improvement. 

To understand the impact of this model of accountability, it is important to know how 
vocational programs present-y function with respect to their multiple constituencies. The 
purpose of this study is to determiae the extent to which local accountability systems exist in 
vocational education and to describe the nature of the underlying relationships between such 
progr. }ms and their constituents. To explore local accoimtability, we visited secondary and 
postsecondary vocational programs in five states: California, Florida, Michigan, Ohio, and 
Oklahoma. We gathered documentary information about programs, and we interviewed 
students, parents, instructors, employers, and administrators. In addition, we interviewed 
staff at the state departments of education. The interviews focused on the relationships 
between the programs and their local constituents. 

Four major conclusions emerged from our investigations. First, there is widespread 
evidence of functioning local accountability systems in vocational programs. Second, these 
accountability systems can be described in terms of four elements — goals, measures, 
information feedback loops, and change mechanisms — and the relationships between them. 
A simple model based on these elements does a reasonable job of describing acx^oimtability 
relationships across a wide range of vocational programs. Third, the quality of these 
components and the relationships between them account for much of the variation in local 
accoimtability systems. Limitatic j in these components interfere with the overall 
effectiveness of the accountability system. Fourth, we can identify many practical 
constraints that reduce the effectiveness of the components in a local accountability system. 
This may provide a basis for developing criteria to evaluate local accoimtability systems and 
prescriptions for improving them. 

These constraints and limitations can be described as deficiencies in the components of 
the model and the interactions among them. Sptjcifically, goals were ineffective when 
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• Higher-level goals were not supported by interlocking goals at the action (local) 
level; 

• Goals were sufBciently broad or vague that it was impossible to know when or if 
they had been achieved; 

• Relevant constituencies did not understand and/or support the goals; and 

• There were unstated or unclear priorities among goals. 

The chief role that measures play in accountability is to provide evidence of the 
attainment of goals, and the most important measures are those that are goal-related. 
Measures were ineffective when they were 

• Not consonant with or sufficient to encompass goals; 

• Of inadequate technical quality; or 

• Not meaningful to constituents. 

Feedback includes the flow of information conveyed by the measures to 
administrators, program staff, and school system constituents, as well as the flow of 
information among administrators and staff. Potential deficiencies in feedback include 

• Insufficient communication; 

• Inaccurate communication; and 

• Low signal-to-noise ratio— :i high r£te of communication with very little useful 
content. 

Ultimately, accountability systems lead to organizational change. Organizational 
reform mechanisms can falter for many reasons, including 

• Regulations that limit options for change; 

• Insufficient resources; 

• Overattention to the needs of one constituency (e.g., employer groups); 

• Giving priority to sh( rt-term demands over long-term trends; 

• Difficulties balancing competing goals and principles (e.g., equity vs. placement); 

• Lack of formal procedures for change; and 

• Ineffective leadership for reform. 
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Greater experience with this model of local accoxuitability will improve its usefulness 
as an analytic tool. For example, the model can serve as a basis for monitoring V-^e effects of 
state and federal initiatives, such as the 1990 amendments to the Carl D. Perkins Act. A 
subsequent report based on this study will explore such policy applications. 
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1. INTRODUCTION 



BACKGROUND 

Over the past several years concern has grown about the competitiveness of the U.S. 
workforce and the efficacy of the U.S. education systems to develop and maintain a skilled 
laboi yoo\. This concern is widespread. According to a recent General Accounting Office 
(GAO) report, "[A]s skill levels are increasing, employers are finding that many young 
workers are inadequately prepared for many entry-level as well as most higher-skilled jobs."^ 
Concern about workforce competitiveness is echoed in the administration's school reform 
proposals as well; two of the six national goals proposed by the administration are directly 
related to workforce productivity.^ 

One of the primary ways in which pohcymakers have addressed this concern is with an 
inci easing emphasis on educational accountability. For exauiple, a major theme of the Bush 
administration's proposal, America 2000, is "creating better and more accountable schools for 
today's students.'*^ Never before have so many diverse groups focused their attention on 
questions of standards and measures of school performance as one means to hold schools 
accountable. Under the umbrella of the administration's initiative to achieve six national 
goals for education, politicians, business people, parents, child welfare advocates, educators, 
and researchers have begun to discuss proposals for establishing national educational 
standards and for developing national tests to assess students' performance relative to there 
standards.^ The intent of these proposals is to create a new framework that can be used to 
monitor the progress of students and to hold schools and school districts accountable. 



^General Accounting Office, Training Strategies: Preparing Noncollege Youth for Employment in 
the U.S. and Foreign Countries, GAO/HRD-90-88, Washington, D.C., May 1990. 

^These two goals are as follows: (1) "American students will leave grades four, eight, and twelve 
having demonstrated competency in challenging subject matter including English, mathematics, 
science, history, and geography; and every school in America will ensure that all students learn to use 
their minds well, so they may be prepared for responsible citizenship, further learning, and productive 
employment, in our modem economy." And (2) "Every adult American will be literate and will possess 
the knowledge and skills necessary to compete in a global economy and exercise the rights and 
responsibilities of citizenship." The White House, America 2000: The President's Education Strategy 
Washington, D.C., April 1991. 

^The White House, 1991. For a wide range of comments on this proposal, see The William T. 
Grant Foundation ComTrussion of Work, Family, and Citizenship and the Institute for Educational 
Leadership, Voices from the Field: 30 Expert Opinions on "America 2000," The Bush Administration 
Strategy to 'Reinvent" America's Schools, Washington, D.C., 1991. 

^For example, see The National Council on Education Standards and Testing, Raising 
Standards for American Education, Washington, D.C., January 1992; W. Johnston and A Packer 
Workforce 2000: Work and Workers fi^r the Twenty^first Century, Hudson Institute, Indianapolis, IN 
June 1987; Science, Education, and Transportation Program, Office of Technology Assessment, 



As concern about workforce competitiveness grows, local systems of vocational 
education aa*e receiving greater attention. Within the educational system, vocational 
educators have been among the most concerned about students^ preparation for work and the 
impact of education on productivity and competitiveness. 

Some might argue that vocational educators have been leading the debate about 
performance-based accoiintability as well. One reason may be that vocational education, by 
its nature, is more ^accountable** than nonvocational education. This is because vocational 
programs have specific outcomes that are conceptually easier to monitor — ^preparing students 
for jobs — and have clearly identifiable constituents — employers — who participate in the 
monitoring function. Vocational education programs have had informal accountability 
systems built around job training and placement for years. 

In addition, federal vocational education and training program initiatives have 
included formal accountability mechanisms. For example, the federal Job Training 
Partnership Act tied resources for traini: ,g directly to job placement and retention through 
the vehicle of performance contracting. Similarly, the 1990 re-authorization of the federal 
vocational education act^ explicitly mandated that states establish standards and measures 
for program performance and the use of these measures to target efforts at program 
improvement. 

The Perkins Act and the reforms it promotes embody concerns about both 
competitiveness and accountability. The act has as its intent "to make the United States 
more competitive in the world economy by developing more fully the academic and 
occupational skills of all segments of the population.*^ Accountability is one of the vehicles to 
be used to accomplish these goals. The Perkins Act requires the development of state and 
local standards and measures, state assessments, and procedures for program evaluation and 
improvement^ These assessment and improvement mandates apply to local districts and to 
states, with most of the final responsibility resting at the state level. It is unclear how these 
requirements will be translated into action or how they will affect ongoing formal or informal 
accountability systems at the local level. 

This project focuses on the juncture of local and state accountability in vocational 
education. Specifically, we are interejsted in xmderstanding local accountability systems in 

Performance Standards for Secondary School Vocational Education, Washington, D.C., April 1989; 
"One Nation, One Curricuium?'* Newsweek, April 6, 1992, pp. 59-60. 

Carl D. Perkins Vocational and Applied Technology Education Act of 1990 (Public Law 

101^92). 

^Ibid., Section 2, 

''Ibid., Sections 115, 116, and 117. 



vocational education, the way they operate, and the data that support tiiem. We also are 
interested in the limitations inherent in existing local systems and the effects of federally 
mandated accountability requirements on these systems. 

The pvesent Note looks at accountability at the local level; a fixture report will explore 
the conjunction of local and state/federal initiatives. The reason we begin at the local level is 
that it is at this level that a competitive workforce is either achieved or not. It is at this level 
that programs are initiated, continued, and canceled; that students enroll, continue, drop 
out, or complete; that skills are learned; and that hiring occurs and workers succeed or fail. 
It is only with data from the local level that we can judge the success of our system of 
vocational education, and ultimately it is at this level that federal and state policies either 
advance or hinder the competitiveness of America's workforce. 

SITE VISITS 

To examii"^ local accountability in greater detail, we visited vocational education 
programs in five stat<.s: California, Florida, Michigan, Ohio, and Oklahoma. The selection 
was based on geographic diversity and the presence of one or more strong accountability- 
related factors, such as a statewide occupational competency system, a set of performance 
standards, or innovative occupational assessment tools. 

Initial contacts were made with the office responsible for vocational education within 
each State Department of Education. We spoke with program staff who were familiar with 
the state's efforts in vocational education assessment and program monitoring and/or 
accountability, and we collected related documents. 

Based on recommendations from the state's Office of Vocational Education and from 
other RAND and National Center for Research in Vocational Education (NCRVE) 
researchers familiar with activities in the state, we arranged to visit at least two local service 
providers. Depending upon the organization of the state's vocational education system, these 
providers were area vocational technical schools, joint vocational school districts, 
comprehensive high schools, vocational high schools, and/or community colleges. 

At each site we interviewed administrators (e.g., the superintendent, principal, 
vocational coordinator, and program coordinator), instructors, students, employers, and, 
occasionally; parents; We spoke with approximately 20 people individually or in small 
groups. We also collected documents describing the school's goals, programs, assessments, 
and accomplishments. 

These interviews and documents permitted us to test and refine our ideas about local 
accountability. The insights we gained from these conversations are reflected in the 
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discussions that follow: we draw upon them for general formulations as well as for specific 
examples* 

OVERVIEW OF THE NOTE 

This Note is divided into five sections. Section 2 provides a brief discussion of the 
nature of accountability in educational systems and describes a conceptual framework for 
accountability in vocational education at the local level. Section 3 presents a detailed 
composite description of a local accoimtability system that one might find in operation in 
vocational programs aroimd the country. Section 4 discusses the limitations in practice tliat 
effectively prevent local accountability systems firom mirroring our conceptual model. 
Section 5 presents a summary of our interim results and conclusions. 
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2. THE NATURE OF ACCOUNTABILITY IN EDUCATION 



Accountability in education refers to the practice of holding educational systems 
responsible for the quality of their products — students' knowledge, skills, behaviors, and 
attitudes. Accountability is neither a new idea nor a new practice in education. Kirst^ 
provides a brief history of its origins, tracing the use of the concept in education to the 
"payment by results" system found in 19th century England. With this system, schools were 
paid according to the performance of their students on standardized exams. 

A "payment by results" system has an internal mechanism that works to optimize 
performance. In its purest form, when teachers are paid directly by their students to achieve 
a particular result (e.g., piano skills, SAT scores, ballet virtuosity), the process operates in 
the following manner. If a teacher cannot satisfy students— because he or she either is not a 
good teacher or is teaching the wTong subject matter-^e teacher fails to make a living and 
leaves the system. As a result, only those teachers who teach the desired subject matter to 
the satisfaction of th^^ir students remain, and the overall quality of the system is improved. 
This is a simple illustration of the mechanism by which an accountability system works to 
optimize performance. 

In the United States, educational accountability has roots in "cost accounting," a 
process for quantifying learning outcomes and attaching costs to them. For example, just as 
a factory can have an objective production goal for manufacturing automobiles, a school can 
have objective "production" goals for education. Similarly, as one can determine the cost of 
producing an. automobile, so one could determine the cost of "producing" a trained graduate,^ 
With this information, it would be possible to compare the output efficiency among school 
systems or among the schools within a system. As we have come to applv it in education 
today, accoimtability reflects this much more complex notion. 

A formal definition of accoimtability can be foimd in recent RAND work:^ 

Accountability describes a relationship between two parties in which four 
conditions apply: first, one party expects the other to perform a service or 
accomplish a goal; second, the party performing the activity accepts the 
legitimacy of the other's expectation; third, the party performing the activity 



^M. W. Kirst, Accountability: Implications for State and Local Policymakers, Office of 
Educational Research and Improvement, U.S. Department of Education, Washington, D.C. July 1990 
2lbid. 

>/rr.«^m ^« ^ J. Bonan, Decentralization and Accountability in Public Education, R.4066- 

MCF/EET, RAND, Santa Monica, 1991, p. 35. 
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derives some benefits from the relationship; and fourth, the party for whom the 
activity is performed has some capacity to affect the other's benefits. 

This is a generic definition of accountability that can be applied at many levels. For example, 
the "parties'* in an accountability relationship could be state and federal agencies, school 
districts and state agencies, schools and school districts, schools and teachers, or teachers 
and students. Educational systems have a large number of constituents, ranging from 
students, parents, and local community members to state administrators, child welfare 
agents, and federal polic)miakers, each with a specific interest in the success of the system. 
Each stands in some type of accountability relationship with schools. Which of these 
relationships have the greatest impact on education? Traditional policymaking focuses at 
the state level, but we will argue that local accoimtability networks deserve greater 
attention, especially in the field of vocational education. 

A CONCEPTUAL MODEL OF LOCAL ACCOUNTABILITY IN EDUCATION 

Levin^ described an idealized educational accountability system as a dynamic, self- 
contained set of perceptions and responses: 

An accountability system is a closed loop reflecting a chain of responses to 
perceived needs or demands; an activity or set of activities that emerges to fill 
those demands; outcomes that result from those activities; and feedback on 
outcomes to the source of the demands. The feedback may generate new 
demands or a regeneration of the old ones; in either case, the previous set of 
activities may be modified or remain intact; a new or altered set of activities 
may be modified or remain intact; a new or altered set of outcomes may be 
produced; arid the loop is conpleted again with feedback to the source of the 
demands (p. 375). 

Using Levin as a starting point, we elaborated a model for a local accoimtability 
system for vocational education that has four major components: 

1. Goals (e.g., 90 percent of high school students vnll graduate); 

2. Measures— means for assessing progress toward the goals (e.g., cost accounting, 
comprehensive record keeping); 

3. A feedback loop-rto provide assessment information and constituent input back 
to the system; and 

^H. M. Levin, ^A Conceptual Framework for Accountability in Education," School Review 82(3) 
May 1974, pp. 363^91. 
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4. A systemic change mechanism — ^for reacting to feedback by changing the system, 
as appropriate. 

We will argue that an effective accountability system must include each of these components 
in a viable form. 

Figure 2,1 presents a diagram of our conceptual model of a local accountability system. 
It reflects both the theoretical perspectives presented above and the input we received from 
interviews with local constituents. Federal and state agencies are represented as ext<5mal to 
the accountability system; that is, the parties in a local accountability system are the local 
school system and its immediate constituents (students, parents, and local business 
interests). This conceptual model provides a basis for examining accountability relationships 
in vocational education programs. To understand local accoxmtability according to the model, 
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Figure 2.1— A Conceptual Model of Accountability 



one must understand goals — ^how they are set, communicated, and changed; measures — ^how 
they relate to goals and how trustworthy they are; feedback loops — ^how information is 
exchanged and how constituents commimicate their opinions and judgments; and change 
mechanisms — ^how the educational system reacts to feedback and how it acts to maximize 
goal attainment. 

Both the individual components in the model and the interrelationships between 
components are far more complex than this simple language would suggest. For example, 
constituent feedback to the program includes not only comments but also actions. Students 
who are dissatisfied with their progress *Vote with their feef* by opting to drop out before 
completing their program. Similarly, employers who are not satisfied with the quality of the 
school's output, i.e., the graduates they interview as prospective employees, reft*ain ft-om 
calling the school for additional referrals. Both of these actions constitute feedback from 
constituents. 

Although such feedback does not contain explicit information about the nature of 
program deficiencies, it constitutes a strong signal that deficiencies exist. Furthermore, a 
decline in student or employer participation ultimately has a negative effect on program 
funding (although the timing and precise mechanisms vary from state to state and locality to 
locality). Thus, this type of negative feedback leads directly to strong formal sanctions. In 
contrast, increased demand for classes or graduates leads to rewards in the form of program 
growth (within local funding constraints). These examples illustrate how rewards and . 
sanctions are elements of constituent feedback. 

There also are rewards and sanctions implicit in administrative feedback to 
constituents. Administrators who respond positively to input from constituents encourage 
further commimication and nurture stronger collaborative relationships. In fact, 
administrators who react **correctly** to feedback by making appropriate program changes, 
but who do not respond encouragingly to constituents, may send unintended negative signals 
that suppress further communication. Nevertheless, the model provides a starting point for 
investigating accountability in specific programs and for thinking about tlie way such 
systems are affected by state and federal policy regarding performance and consequenccis. 

LIMITATIONS OF STATE- AND FEDERAL-LEVEL ACCOUNTABILITY SYSTEMS 

Most educational policymaking occurs at the state or federal level,^"' and, as a result, 
most formal accoimtability mechanisms focus on information that is highly aggregated. For 

^See L. M. McDonnell and W. N. Grubb, Education and Training for Work: The Polio/ 
Instruments and the InstUutions, RA02e NCRVE/UCB, RAND, Santa Monica, 1991. 

2 ^ 
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example, for many years the Secretary of Education published a *Wall Chart" that display^^d 
state-level data on educational resources and attainir.ents. Similarly, Congress authorized 
the National Assessment of Educational Pi ccnre":, (NAEP) to conduct trial state assessments 
in 1990 and 1992 to provide achievement comparisons at the state level.^ Formal evaluations 
of large-scale, federal, categorical aid programs'' (e.g., Title I, Title III, and Title VII) drew 
conclusions at the national level that became significant parts of the policy debate on 
reauthorizing these programs, just as the National Assessment of Vocational Education 
produced national results that affected the reauthorization of the federal vocational 
education programs.^ 

Similarly, most research on accountabiHty is focused on ttxe needs of state and national 
policymakers. For example, in a companion study conducted under the auspices of the 
NCRVE, Hill et al.^ examine the information needs of national policymakers and the 
networks through which these needs are met. 

However, there is a potentially serious problem in basing policy on accountability 
measured at highly aggregated levels. Mechanisms that appear to support policymaking at 
state and national levels may result in policies that are inappropriate when implemented at 
the local level. A simple example of this would be policymaking based on statewide 
occupational demand data. State policies that require vocational schools to offer or to 
withdraw occupational training programs based on statewide occupational emplcfyment data 
may be insensitive to local variations in workforce needs. For example, shortages of medical 
support personnel such as nurses or respiratory technicians can be highly localized, so it is 
important to base decisions about program initiation or termination on conditions in specific 
areas rather than on average conditions statewide or nationwide. 

Vocational education i ticularly susceptible to ineffective, high-level policymaking 
because local conditions play such a large role in program planning and performance. 

^In fact, the sampling plan for the original National Assessment of Educational Progress was 
specifically designed to preclude reporting of results at the state or district level. See S. Messick, A 
Beaton, F. Lord, National AasessTnent of Educational Progress Reconsidered: A New Design for a New 
Era, NAEP Report 83-1, National Assessment of Educational Progress, Educational Testing Service, 
March 1983. 

"^For example, P. Herman and M. W. McLaughlin, Federal Programs Supporting Educational 
Change, Vol VIII: Implementing and Sustaining Innovations, R-1589/8-HEW, RAND, Santa Monica, 
1978. 

®J. G. Wirt, L. D. Muraskin, D. A Goodwin, and R. H. Meyer, Final Report, Volurne /, Summary 
of Findings and Recommendations, National Assessment of Vocational Education, U.S. Department of 
Education, Washington, D.C., July 1989. 

T. Hill, J. Harvey, and A Praskac, Pandora*s Box: Accountability and Performance 
Standards in Vocational Education, R-4271-NCRVE/UCB, RAND, Santa Monica, 1992. 
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Programs are planned with an eye tx)ward local economic needs, and program performance is 
measured in terms of placements, which are found primarily in the local area. 

Local business and industry representatives play a significant role in many aspects of 
vocational programs, giving each program a imique local profile. For example, local business 
and industry help industrial vocational programs define goals and determine curricxilum. 
Employers serve as advisors to almost all occupational-specific training programs. In this 
capacity they help programs identify skill needs, target training, and provide direct links to 
employment. Furthermore, many vocational instructc>rs have direct contact with employers 
on a regular basis. Instructors are required to maintain their occupational skills and to be 
aware of the evolving state of the art in their fields by spending time at local businesses. 
Instructors also interact with employers when they supr^ ,ise students working in job 
settings as part of training programs. 

Vocational programs are responsive to the business commimity because they depend 
on local business for placement. Employers review the output of vocational programs (in the 
form of potential new employees) and make judgments about its quality (in the form of hiring 
decisions). In all these ways local business and industry affect vocational education, and 
poHcymaking needs to be sensitive to these influences. We believe it is important to 
understand local accountability mechanisms and their relationship to more highly 
aggregated state pohqyonaking. 

A final reason to focus on local accountability systems is that such mechanisms are 
likely to lead to program improvement more easily than highly aggregated ones. As Levin^^ 
noted, an accountability system is more effective when linkages are tight and information is 
produced and shared freely. These conditions exist more often at the local level than at the 
state or federal level. 

We conducted site visits to vocational programs in five states. These visits confirmed 
our initial impressions that local accountability systems exist, although they are often 
informal and unsystematic. Furthermore, the visits revealed a number of commonalities 
across sites. 

In the following section we use the insights gained firom our site visits to describe a 
local accountabihty network in a hypothetical vocational technical school. The purpose of 
this generic description is to illustrate the manner in which the abstract relationships from 

i^Levin, p. 375. 
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the model are acted out in a real setting. By fleshing out the model in this way, we both 
enliven the text and elaborate the relationships. This elaboration is useful in later sections 
when we attempt to describe some of the limitations of local accountability systems. 
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3. A GENERIC EXAMPLE OF LOCAL ACCOUNTABILITY IN VOCATIONAL 
EDUCATION: MIDAMERICA VOCATIONAL TECHNICAL SCHOOL 



INTRODUCTION 

Our pxxrpose in describing a generic vocational technical school is to illustrate the 
elements of the local accountability model described in the previous section. Schools are 
extremely complex organizations, involving diverse sets of participants, functions, facilities, 
regulations, and beneficiaries. Of necessity, this description will omit much that is not 
essential for describing the accountability components and relationships of interest, i.e., the 
goals, measures, feedback loops, and change mechanisms at work in the setting. 

MidAmerica Vocational Technical School (MAVTS or MidAmerica) is our own creation; 
to our knowledge there is no such school in the United States. We created MidAmerica to 
exemplify a "good" vocational technical school, one that serves its constituents well. MAVTS 
provides high-quality training programs for widely diverse students, and it appears to meet 
the needs of its constituents and the needs of the regional commimity for employment 
training. However, the school is not wholly imaginary; it represents a composite of 
programs, people, and practices we observed in vocational schools in the United States. 
Based on our observations, we are confident there are many institutions of equal or better 
quality in this country. 

The success of MidAmerica can be traced to many factors: a dedicated staff, an 
informed and sufportive commimity, a cooperative state vocational education department, 
and a well-grounded curriculum. However, the feature that links these factors together and 
keeps MidAmerica ^'on target** is a rich, multilevel, local accountability network. 
Accountability at MidAmerica is based on a collection of simple components widely present in 
vocational education programs at the secondary and postsecondary levels. The 
accountability system is not a rigid set of data elements, reports, and processes, but a diverse 
collection of measures and commimication paths through which all of MidAmerica's local 
constituents— employers, community members, students, instructors, administrative staff- 
make their needs and opinions known and in response to which program staff act to improve 
the programs offered by the school and to be responsive to the needs of students and 
employers. 

The description of the school will be divided into five parts: a one-paragraph 
''snapshot,'* and lengthier sections describing district-level features, school-level features, 
program-level features (including examples of two typical programs), and the relationship of 
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the school to state and local constituents. Following these discussions, we will describe the 
local accountability system as it functions at MidAmerica. This description will cover goals, 
measures, feedback loops, and mechanisms for organizational change. 

SNAPSHOT 

MidAmerica serves over 750 fiill- and part-time students; approximately one-half are 
high school students and one-half are adults. The school offers certificates of mastery in 
more than 30 occupational areas. All programs are competency-based and use the state 
occupational competency task lists as the basis for their curricula. Most programs are 
organized on an open-entry/open-exit basis. The school receives the bulk of its funding from 
local taxes but also receives substantial funding from the state and some funds from federal 
sources. Most of the buildings and facilities are over 30 years old, although many have been 
modernized as new programs were implemented. The Office of Vocational Education in the 
State Department of Education considers MidAmerica to be one of the better vocational 
technical schools in the state. 

DISTRICT FEATURES 

As with many schools, MidAmerica is part of a school district that includes a network 
of elementary, secondary, and vocational schools. In this section, we describe several salient 
features of the school district that have either a direct or indirect effect on local 
accountability. 

Governance 

MidAmerica is one of four area vocational technical schools operated by the 
Southwestern Regional Training District, a state educational entity that spans two counties. 
The district has the power to levy taxes to support secondary and adult vocational training. 
It also must comply with all state regulations governing secondary schools, including 
accreditation, licensing, and facilities. 

The district is run by a Board of Education, elected by the citizens. The board sets 
policies for the four vocational schools under its jurisdiction. The board appoints a 
superintendent to supervise the operation of the district. The current superintendent. Dr. M. 
Blanhom, has been in office for seven years. Dr. Blanhozn recommends the appointment of 
principals for each school. The superintendent, principaU', and all school staff are employees 
of tlie Southwestern District. 

Since taking the position of superintendent. Dr. Blanhom has encouraged individual 
schools to be responsive to the needs of their local communities, and he has given each of the 
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four school principals moderately wide latitude to implement the policies of the board in 
ways that are appropriate to its local service ^ ^^a. 

Mission 

The mission of the Southwestern Regional Training District is contained in a formal 
statement, adopted initially by the board in 1985 and reviewed most recently in 1990. It 
commits the district to a mmiber of specific purposes, including the following: 

• Preparing students to find meaningfial emplo3rment, with the potential for long- 
te'xu careers; 

• Creating a well-trained pool of potential workers that meets the current and 
future needs of the regional economy; 

• Helping business and industry upgrade the skills of the existing workforce; 

• Preparing students to obtain advanced occupational training in community 
colleges or other postsecondary training institutions; and 

• Meeting the needs of special populations, such as students with disabilities, 
students who are re-entering the workforce after many years' absence, and 
students from minority population groups. 

The district defines its mission in terms of service to students and the community. 
Furthermore, it does so in ways that usually are measurable — ^information can be gathered to 
determine whether the district is achieving these purposes. The mission statement is shared 
with all staff and students, and it is a prominent part of the district's institutional culture. 
Furthermore, each school is reqxiired to adopt a set of goals that are consistent with the 
district's mission, and these goals are generally imderstood as the key elements that define 
each school's purpose. 

Funding 

Approximately 60 percent of the operating costs of the district are raised by local ad 
valorem taxation. Taxing authority must be voted on by the citizens in the region every 
other year^.and the district has received continuing commimity support. The state provides 
another 30 percent of the school's revenue in the form of program-based funding and special 
grants. Federal vocational education fimds distributed by the state account for an additional 
7 percent of the budget. The remainder of the annual revenue is made up of student fees. Job 
Training Partnership Act (JTPA) program fimds, and miscellaneous sources of revenue. 
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Services to Schools 

The district provides only limited administrative services for the schools, leaving most 
administrative functions in local hands. The centralized services include accounting, pa3rroll, 
employee benefits administration, and legal support. 

SCHOOL FEATURES 

Although much of the external context within which the school operates is set by the 
district, it is at the school level that philosophies and mission statements are translated into 
action. It is also at this level where the basic information that feeds the accountability 
system is collected In this subsection, we briefly discuss the characteristics of the school 
that directly or indirectly affect accountability. 

Administration 

The authority for operating the school resides with the principal, Dr. S. Monet. She is 
responsible for most school functions, including hiring and firing of staff, developing and 
administering the budget, and initiating and supervising programs. The superintendent 
intervenes only when he perceives there is a problem that is not being addresser? adequately 
by the principal. 

Goals 

MidAmerica translates the district's mission statement into specific programmatic 

goals: 

• To assess the needs of students for education in basic academic skills and to 
provide basic skills training as needed; 

• To provide appropriate, relevant, up-to-date occupational training for students; 

• To provide appropriate, short-term training to existing workers to upgrade their 
skills and meet the needs of a changing workplace; 

• To use the expertise of local business and industry in program development; 

• To help students find training-related jobs in the region; 

• To meet the needs of local business and industry by improving the skills of the 
workforce; and 

• To offer specialized training programs for students with special needs. 
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Funding 

Although the bulk of MidAmerica's revenue comes from local taxes, the state provides 
approximately 30 percent of the school's total fundint through the Office of Vocational 
Education (OVE) of the State Department of Education. The distribution of federal 
vocational education funds also is directed by OVE, State funding is provided on a per- 
program basis rather than a per-student basis; i.e., a fixed amount of funding is awarded to 
each authorized program with the intention that the program will serve at least 20 students. 
However, the funding level does not change if enrollment changes during the yeai* 
(Enrollment is one element that is considered in the state's program review, and continued 
low enrollment can lead to a cutoff in state funds. See below.) 

Students 

MAVTS has an enrollment of 760 full- and part-time students (divided approximately 
equally between adults and concurrently enrolled high school students). Approximately 25 
percent of the high school students and 15 percent of the adults are members of minority 
population groups, and 5 percont of the adults are women re-entering the workplace after 
many years. In addition, MidAmerica has a special program for deaf students, which draws 
students from throughout the district. Currently, 18 hearing-impaired students receive 
special services under the auspices of this program. 

Contralized Support Services 

Most of the responsibility for instructional services is delegated to individual 
programs, but some common support services are provided centrally. These include outreach 
and recruitment, initial assessment and counseling, and the Learning Center. 

Outreach and Recruitment. TLz school has an active program of outreach to local 
high schools and the community. In addition, staff produce printed brochures and radio 
public-service announcements at the school and use these to inform potential students of the 
programs that are offered. Once each year, staff members make presentations at each 
secondary school in the region as part of a "career evening^ sponsored by a local business 
association. 

Counseling andlnitial AssessmenL All prospective students receive an 
appointment with a career counselor. The counselor discusses the student's interests and 
reviews the opportunities available at the school. The counselor also arranges for the 
student to undergo an hour-long assessment of academic skills and vocational interests. This 
information is shared with students to help them make better choices and to alert them to 
special services that are available at the school, such as the Learning Center, which provides 
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instruction in basic skills. The counseling center staff also can arrange for a student to visit 
classes for a brief period to familiarize the student with career options in that field and to 
assist the student in choosing among training opportunities. 

Learaing Center. The Learning Center is a recent addition to MAVTS, created in 
response to the growing number of students entering the school with deficiencies in basic 
literacy or quantitative skills. The center provides specialized instruction in basic skills and 
coordinates the delivery of this instruction with the student's regular occupational program. 
Students who lack the basic skills necessary to complete their chosen program enroll in the 
program so long as they continue to work on basic skills in the Learning Center. The center 
teachers try to coordinate their instructional program with the occupational training the 
student is receiving. 

Industry Education. In recent years MidAmerica has actively promoted its 
retraining capabilities. The Industry Education i'yjzr^ helps individual programs develop, 
organize, and administer short-term courses designed to meet the needs of local businesses 
for upgrading the skills of existing workers. Often these courses are held at the job site. Tl-ie 
Industry Education Office makes all necessary arrangements so programs do not have to 
worry about facilities, scheduling, enrollment, or credits. Unless funding is provided by other 
governmental sources, such as JTPA, the ttiition is set at a level that permits the school to 
cover its costs in oflFering the course. 

PROGRAM FEATURES 

Programs are the most important administrative imits at MAVTS. Decisions 
regarding curriculum and instruction are made at the program level, and it is at the program 
level that the school has the greatest contact with students, parents, and local businesses. 
For example, feedback from businesses regarding the appropriateness of training content is 
received and acted upon primarily by programs. MAVTS has 10 departments that offer 34 
different occupational training programs. For example, the graphic commimications 
department offers programs in advertising art, offset printing, composition, and production 
art. 

Organization and Scheduling 

All full-time programs are offered on an open-entry/open-exit basis, and, as a result, 
instruction is highly individualized. Concurrently enrolled high school students attend for 
three hours per day ai:d earn academic credit. Adult studer.ts can atte^.d for either three or 
six hours per day. For unlicensed occupations, approximately 600 course hours are required 
to complete competencies and earn a certificate issued by the school. For licensed 
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occupations, such as cosmetology, state requirements for length of training apply, and 
students must pass a state examination to earn a license. 

Short-term Industry Education *;ourses are not open-entry/open-exit; they have a fixed 
length, but they operate on an ad hoc s?liedule. Course lengths are determined in advance 
based on the content to be covered. 

Competency-Basod Curriculum 

All courses, both short-term and full-time, are competency-based. This means their 
curricula are divided into sequences of discrete elements, each consisting of knowledge to be 
learned and/or skills to be mastered. The state OVE supervised the development of skill lists 
for each occupation that are used as the basis for the curriculum. State skill lists are 
supplemented with additional skills suggested by local employers. Student progress can be 
monitored in terms of the number of competencies completed. 

Measuring Student Performance 

Instructors are responsible forjudging student performance. Each instructor 
determines the method of assessment to be used for each competency: written tests, 
coursework, or actxial performances. Dr. Monet, the principal of MAVTS, encourages the use 
of hands-on performance assessment whenever appropriate. OVE also has developed pendl- 
and-paper mastery tests linked to the occupational skills lists; these are used by some 
instructors to measure competence. Programs maintain individual records and issue 
certificates of mastery when students complete all competencies required for an occupation. 
In addition, high school students receive course credit based on attendance and satisfactory 
performance, whether or not they achieve a certificate of mastery. 

EXAMPLES OF PROGRAMS 

In this subsection we briefly describe two programs, one from the 
industrial/mechanical department and one from the cosmetology department. 

Automobile Service Technology 

The automobile service technology program has been in existence since the school was 
opened, and it is one of the most -''ell-established programs offered at MidAmerica. It has 
well-maintained, up-to-date equipment, and the instructors are well trained. In fact, the 
program holds Uie ASE Master Certification from the Automotive Service Excellence 
organization in six areas, and four instructors also hold individual Master Certification from 
ASE. 
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The program comprises the following six elements: automotive core (shop safety, tool 
use and identification, engine parts identification and theory of internal combustion engines), 
automotive systems (electrical, electronic, brake, air conditioning, transmission, engine, 
suspension, and fuel), autotronics, front end and brake repair, transmissions, and air 
conditioning. 

Program graduates have found jobs as technicians in local automobile dealerships and 
garages or as self-employed automobile repair technicians. Postsecondary students can 
receive credit toward an associate degree through a cooperative agreement with Audubon 
Community College. 

Cosmetology 

The cosmetology program began three years ago. The program operates a cosmetology 
salon on the school grounds that is open to residents of the community. The cosmetology 
curriculum includes hair styling, shampooing, permanent/finger waving, hair coloring, 
facials, scalp treatments, makeup, manicuring/pedicuring, and nail sculpturing. Each 
student learns through classroom instruction and has an opportunity to apply this knowledge 
to hands -on training in the cosmetology salon. 

The cosmetology program follows the regulations established by the State Board of 
Cosmetology. To qualify to take the state board examination, students must complete 1200 
hours of training. Upon completion of the training, students must pass the state 
examination to obtiin a license to work in the field. 

Cosmetology is a profession that is very individualized; a cosmetologist works 
independently in the development and formation of the business. Program graduates have 
found jobs in salons, department stores, and cosmetic firms as cosmetologists, hair stylists, 
nail technicians, makeup artists, or managers. 

RELATIONSHIP OF MAVTS TO STATE AGENCIES AND LOCAL CONSTITUENTS 

The relationships that a vocational school has with governmental agencies and local 
constituents (i.e., students, parents, businesses) may be characterized as an exchange of 
resources and information. The State Department of Education (SDE) provides vocational 
education funds to the school districts and establishes regulations governing the use of the 
fxmds. School districts, in turn, must comply with the regulations and must provide reports 
back to the state documenting compliance and program effectiveness. Ir. this subsection, we 
provide a flavor of these relationships for MAVTS. 
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State Departm«nt of Education and Other State Agencies 

The State Department of Education through its OVE provides a number of resources 
to region^ vocational technical schools. The state also develops regulations that govern the 
operation of these schools. 

Program Development. The Office of Vocational Education provides a set of state- 
level job skill lists for each of the occupations covered by vocational programs in the state. 
Employers representing all regions of the state participated in the development of these skill 
lists to ensure that they were valid reflections of the demands of the occupations. These 
skills lists are available to any vocational program in the state to use as a basis for 
developing curriculum and certifying student preparation. 

OVE also develops competency-based assessment tools that correspond to the 
occupational skill lists. Most of these assessment materials are pendl-and-paper tests, but 
some are performance-based assessments. Again, these tests are available to vocational 
programs in the state to use as they see fit (Initially, the tests were to be mandatory, but 
this requirement was dropped after significant objections were raised in the state 
legislature.) 

Program Review and Evaluation. Programs cannot receive state funding xmless 
they have been authorized by the Office of Vocational Education, and this authorization must 
be renewed every two years. Reauthorization is based on continued labor market demand 
and on evaluations of program quality. If OVE determines that there is a need for training 
in a particular occupation and that the program is qualified to meet the need, it will 
authorize the program to provide one or more classes of instruction in the occupational area. 
Each authorized class receives a lump sum annual allocation of fimds. The state assumes 
that, on average, an authorized program will serve the equivalent of 20 students per class. 

The OVE conducts biennial program reviews that are the basis for continued program 
authorization. The reviews focus primarily on three factors: the quality of the program, the 
level of demand for the occupation, and the success of the program in finding jobs in related 
fields for students who complete the program. The reviews also consider employer 
satisfaction and community support. 

Data Systems. The Office of Vocational Education conducts follow-up surveys of 
program graduates six months and one year after they finish their program. These surveys 
collect information on current employment and educational status, length of employment, 
and the relationship between employment and training. A supplemental survey attempts to 
measure employer satisfaction. These data are summarized and reported to the school 
annually. 



.21- 



Coordination of Economic Development Programs. The State Department of 
Labor produces semi-annual labor market projections that schools can use to estimate 
demand for jobs. Unfortunately, the data represent statewide demand, and there is no 
attempt to estimate demand within particular regions. Over the years schools have become 
reasonably adept at making local projections based on the statewide data, and most continue 
to use the statewide data. 

Efforts are imder way to create regional planning consortia, including all agencies and 
contractors providing occupational preparation and training services. This would include 
Private Industry Councils in each JO PA service delivery area, proprietary schools, adult 
education programs, community colleges, regional vocational technical schools, and volunteer 
literacy programs. To date the regional consortium in MidAmerica*s region has met only 
once to ^'get acquainted,"* and it is too early to tell how useful this forum will be. 

Local Employers 

MidAmerica has both formal and informal contacts with the local business community. 
Each program must have an industry advisory committee consisting of employers and 
representatives of trade associations and occupational groups. The advisory committees at 
MidAmerica have between eight and fifteen members. Committees meet at least twice a year 
to review curriculum, instructional equipment and materials, training approaches, and 
standards. Furthermore, Dr. Monet encourages programs to involve committee members in 
judging actual performances of students approaching pro-am completion. 

Instructors are responsible for student placement (it is one of the criteria that are used 
to judge instructor performance), so they try to maintain good relations with local employers. 
Most instructors are on a first-name basis with the owners of small businesses in the area. 
In addition, they have good relationships with larger employers (sometimes with staff in the 
personnel office but more often with managers in related occupational depail^nents). As a 
result, employers, particularly small businesses, often contact instructors at MAVTS directly 
when they have job openings. 

Instuctors also maintain direct contact with local businesses as part of their 
professional development. Each must spend 10 hours per term observing at local job sites. 
This on-the-job time helps instructors maintain their professional skills and their 
understanding of the needs of local employers. 

Employers also are involved in MidAmerica's Industry Education Office program. 
Industry Education provides short-term classes to upgrade the skills of currently employed 
workers. These programs are developed in coryunction with local employers and employee 
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groups, who D^'ost help the school prepare the curriculum outline and develop the list of 
competencies. The number of Industry Education courses has been increasing rapidly over 
the past two or three years. As a result, there have been increasing contacts between the 
school and local businesses. 

Students and Parents 

The opening of MidAmerica was accompanied by great fanfare, and it generated much 
local interest. Initial enrollments were high. However, after several years enrollments 
began to decline, and the school initiated outreach and recruitment efforts aimed at high 
school students and adults. These efforts were successful, and they have been continued to 
the present day. 

Most high school students learn about MidAmerica through presentations at their 
school, recommendations from counselors, or printed material distributed at the school. Most 
adults learn about the school from brochures distributed by state agencies and private 
counseling groups, advertisements on radio or billboards, or word of mouth from their 
friends. The outreach programs to high schools have been particularly successful, and there 
are often waiting hsts for some programs. Adult enrollment is increasing as well. 

MidAmerica has limited contact with parents. Despite efforts to engage parents of 
high school students in discussions about career plaiming and training, very few parents 
actually go beyond the front door of the school. 

ACCOUNTABILITY MECHANISMS 

It is easy to describe on paper the way the organizational units are supposed to work 
together to achieve the schooVs goals, but it is harder to make this happen in practice. 
Although Dr. Monet and the staff might not recognize it as such, MidAmerica has an 
effective local accountability system that helps them stay on target. All the elements of local 
accountability are present to some degree at MidAmerica— goals, measures, feedback loops, 
and change mechanisms. They are reflected in the way various decisions are made at the 
course, program, and institutional levels. The following material highlights specific 
accountability-related features at MidAmerica. 

Goals 

Dr. Monet talks about her "expectations'* regarding the activities each organizational 
unit must engage in so the school as a whole can achieve its goals. These expectations are 
really Dr. Monet's way of operationalizing goals for each unit, beginning with the 
administration itself. 
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Administrative Goals. The school administration is expected to provide centralized 
individual student assessment services and training for students needing to improve basic 
skills. It also is expected to conduct outreach and recruitment activities in the community, 
making students (both full-time and short-term) aware of the opportunities the school 
provides. The institution is expected to support the programs in their efforts to build better 
links to local employers and the commxmity, to act as channels for information and support 
provided by the Office of Vocational Education of the State Department of Education, and to 
provide adequate facilities to meet the needs of programs. 

Program Goals. Dr. Monet holds each program responsible for helping MidAmerica 
achieve its overall goals (see above). Although the school's goals are stated in general terms, 
they have specific implications at the program level. Each program is responsible for the 
following: 

• Helping students master appropriate job skills in their occupational area as 
delineated on the occupational competency lists; 

• Providing students with improved employment skills, such as personal 
deportment, timeliness, effective work habits, and teamwork; 

• Helping students find employment in jobs related to their training; 

• Ensuring that the program articulates with advanced training options at local 
community colleges, when appropriate, and encouraging students to pursue these 
advanced alternatives; 

• Providing short-term training in job-related skills for employees of area 
businesses; and 

• Adapting instructional materials and conditions to meet the needs of students, 
referring students to appropriate MidAmerica support services such as the 
Learning Center and counselors, and helping students with special needs find 
employment opportunities by working with employers to develop appropriate 
work options. 

These goals are operationalized further in specific course- and program-level 
objectives, including the following: 

• To enroll adequate numbers of students; 

• To provide support for students so they complete training; 

• To help students find employment in a training-related occupation; 
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To maintain the quality of the equipment and the curriculum; and 
To make the training responsive to the needs of the workplace. 



In Dr. Monet's terms, programs are expected to demonstrate that they are meeting a 
real employment need in the region. They are expected to maintain lists of the skills and 
abilities needed to perform each job for which they offer training and to base their training 
on these occupational competencies. Occupational skill lists are expected to be updated 
regularly and to reflect both local employment conditions and industry standards for 
comparable positions. Programs are expected to align their curriculum with their skill lists, 
and they are expected to develop an assessment system to certify mastery of skills as 
students progress through courses. Finally, they are expected to help students find 
employment or further training; in areas related to the program. 

Instructors in each program are responsible for meeting the program's expectations. 
They are expected to provide relevant learning opportunities for all students enrolled in their 
courses, and they are expected to coordinate their efforts with the school's Learning Center. 
Instructors are expected to build positive relationships with local employers and trade 
associations, both for the purpose of student placement and to upgrade their own skills to 
keep pace with changes in the industry. Perhaps most important, they are responsible for 
helping students locate employment. 

Goals for Students. From the institution's perspective, the most important student 
goals are the following: 



Specifically, students are expected to attend courses regularly, complete assignments 
in a timely manner, seek assistance in improving basic skills at the Learning Center if 
needed, and conduct themselves in an appropriate manner. They are expected to take 
responsibility for managing their education and employment preparation, monitoring their 
mastery of competencies, and calling the instructor's attention to their needs. 

In most cases, students hold the same goals for themselves as the institution does: to 
improve their job-related skills in preparation for meaningful employment and careers. 
However, not all students share these goals. For example, some secondary students are 
merely exploring career options; they are not interested in immediate employment upon 



To learn necessary academic skills; 

To learn specific job-related occupational skills; and 

To find employment and/or additional employment-related training. 
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completion of high school. Other secondary students are enrolled only to get coiirse credit 
toward graduation; they are not motivated to learn the material in the courses nor to master 
occupational skills. Despite MidAmerica's efforts to educate schools and coxmselors about the 
purpose of the school and the characteristics of successful students, some schools continue to 
"dump" unmotivated, low-achieving students onto MAVTS without proper preparation or 
counseling. MidAmeriea tries to accommodate the needs of students who are still exploring 
career options or whose goals are not immediate employment, but it does not see this as its 
chief mission. 

Similarly, there are some adult students (though the numbers are small) who enroll at 
MidAmeriea for very different reasons. Some adults have multiple problems: they lack basic 
skills, they have gone through traumatic family or personal experiences, or they have been 
xmsuccessful in attempts at work and they need counseling and assistance in a number of 
ways. They hope MidAmeriea will solve all their problems, but it may not offer all the 
services they need. Finally, a few adults 'ire enrolled because they are required to do so to 
continue receiving public assistance. MidAmeriea recognizes that there are differences in 
adult students' goals, but it places its chief emphasis on the more traditional job-oriented 
adult. 

Industry Education Office programs are mors narrowly focused on a specific set of 
skills. They assume that students are motivated to master skills that improve their standing 
at work, to qualify for new positions, or to keep abreast of changes in the workplace. 

Goals for Employers. MidAmeriea anticipates that employers will: 

♦ Help the school identify training needs and establish training programs; 

♦ Provide advice to programs regarding current job skills, working environments, 
and demands on workers; 

♦ Consider program graduates as potential employees; 

♦ Offer short-term internship or work-study opportunities for students; and 

♦ Use the school as a resource for upgrading the skills of existing workers as the 
demands of the workplace change. 

All employers are encouraged to participate in the program in some manner. Those 
who agree to serve on advisory committees are expected to meet regularly, to review the 
program curricula and skill lists, and to update them to match local needs and industry 
standards. Those who agree to help assess student performance are expected to attend 
review sessions and to observe, evaluate, and critique student work products and 
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performances. Those who arc interested in hiring students who complete the program are 
expected to provide feedback to the program about the performance of graduates. 

Employers' immediate goals are to find qualified employees to fill current openings. 
However, they also are concerned about the long-term needs of their industrial sector. They 
often look beyond the demands of one store or factory to the ongoing needs of an industry and 
are willing to help the school take this long-range perspective. However, employers are not 
always willing to put in the time in an advisory role that the school desires, nor do they 
always maintain a long-range perspective. 

Measures 

MidAmerica's accountability network functions reasonably well because measures 
exist that are relevant to many of the goals. Although there is not a one-to-one match 
between goals and measures, there are considerable data that can be used as indicators of 
the school's success in meeting many of its goals. There remain some school goals that are 
not easily assessed, as well as some measures that do not relate to a goal deemed to be 
important to the school. (In the next section we discuss the problems that occur in these two 
situations.) 

Measures of Program Effectiveness. Programs collect a wide range of information 
related to course and program objectives in the areas of enrollment, completion, and 
placement. 1 Measures relating to student and employer goals are discussed in the following 
text. Aggregated student data provide the chief measures of program success in these areas. 
For example, each program can track course enrollments and completions and can ^renerate 
estimates of the percentage of students attending and completing individual courses or 
course sequences. They also can summarize program success in terms of students' grades 
and can report on the percentage of students in a course who have mastered a particular skill 
or collection of skills. 

Programs try to keep records of initial placements in training-related jobs, but they 
lose contact with many students (for example, those who move out of the area) and the data 
have always been incomplete. The chief measure of post-training activity is the OVE follow- 
up survey described above. 

In addition to data on enrollment, completion, and placement, programs also keep 
track of information related to community contact and to program quality. Data are retained 
describing the number of inquiries each program receives fi-om potential students and fi-om 
employers looking for new employees. Programs also maintain anecdotal records of contacts 



^The most important of the program goals are listed on page 23. 
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instructors have with high schools, community groups, and other social service agencies. 
These activities are important for recruitment, placement, and maintaining a positive image 
in the community. 

The school does not collect any direct measures of instructor quality. Quality is 
inferred from student success, employer satisfaction, and participation in continuing 
education. The administration uses aggregated student performance data and employer 
satisfaction data as ways to judge overall program quality. The only direct measures of an 
instructor's behavior are reports on employer visits designed to maintain an instructor's own 
skills and knowledge of the workplace. 

Measures of Student Progress. A variety of data is available to measure student 
progress, and most are directly relevant to the program's goals for students — attainment of 
academic and occupational skills and employment in a training-related job (or enrollment in 
additional training). 

Each instructor is responsible for tracking student performance against the course 
skills list and for certifying students' mastery of ii^dividual skills. Instructors are expected to 
match the method of assessment to the nature of the skill. Thus, content knowledge can be 
assessed using paper-and-pencil tests, but procedural knowledge should be assessed through 
performances. Ultimately instructors are responsible for determining whether student 
performance on a skill is adequate. Some programs hav^ opted for commercially' developed 
standardized performance assessment, but most use their own tasks and examinations. 
Some programs use the state competency tests as "end of program" tests to confirm skill 
mastery. Instructors also are encouraged to involve local employers in judging large-scale 
student projects/activities, and this is done quite often in some programs. 

Each program maintains files on individual students containing information about 
their progress in individual courses and in the program as a whole. The files include 
information about attendance, course grades, and occupational skills mastered. The 
Learning Center also maintains individual records on student attendance, skill attainment, 
and overall achievement in basic skills. 

Monitoring the occupational status of students after they leave the program is a more 
difficult task. Instructors are asked to keep track of the initial employment of all students 
who complete a program, and they are able to do this for the majority of students. However, 
they have little or no information about those who leave before completing the program, and 
data on initial employment are seldom verified. 

The main source of information about employment or continued training is follow-up 
surveys conducted by the state Office of Vocational Education. The school supplies OVE with 
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information about students who complete the program and C^rE tracks their employment 
and educational status at six and twelve months after completion. The results of these 
surveys are shared with the school, and they are the major source of data about student 
employment and training. OVE surveys collect information about current employment 
status, relationship of training to employment, additional training received, and satisfacti*. n 
with training. 

Measures of Employer Participation and Satisfaction. The most direct measure 
of employer satisfaction is the employer follow-up survey conducted by OVE as part of its 
student follow-up survey efforts. Employer satisfaction also is measiu*ed indirectly in terms 
of continuing participation in program activities. Local industries are asked to participate on 
industry advisoiy committees, to participate in judging student performance, to provide 
opportunities for cooperative work while students continue their training, and to employ 
program graduates. Each program keeps careful records of employer contacts in all four 
areas as well as any other ways local businesses are involved in supporting the programs, 
such as donating equipment, sponsoring events, and permitting tours of the workplace. 

Feedback Loops 

Administrators and staff receive feedback about program performance from a variety 
of sources and in a variety of ways. Some feedback is formal (including the results of OVE 
student follow-up surveys, students' scores on competency tests, and reports of advisoiy 
committee program reviews). Some is informal, such as comments on students' abilities from 
businesses calling to look for prospective employees. Some feedback is internal to the system 
(e.g., data on student completion rates are generated as part of the ongoing administrative 
data system), and some comes from constituents (such as employers' reactions to course 
competencies and parents' opinions of the school's effectiveness). Finally, some feedback, 
such as course completion rates, is provided directly to program administrators, while other 
feedback, such as students' comments on course relevance or job competencies, comes directly 
to instructors and other staff. Overall, there is a rich network of information from a variety 
of sources that can be used for program evaluation and improvement. 

Feedback Regarding Program Effectiveness. The school receives a formal report 
from OVE after each biennial program review. Among other things, this report reviews the 
need for and performance of each occupational program and contains recommendations about 
the continuing authorization of the program. In addition, the Board of the Southwestern 
Regional Training District reviews the school's performance axmually as part of its oversight 
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responsibility and makes its opinions known through policies and directives as well as 
through its endorsement of the superintendent 

Similarly, MidAmerica's Ad\dsoTy Coxmdl conducts a formal annual review of the 
school's performance. The Advisory Council reviews each program area. Industry 
participants on the individual occupational advisory committees have direct input into 
curricultim, materials, and program planning. 

Informal feedback comes in a number of ways. The citizens of the district express 
their judgments regarding the value of the schools by voting for the members of the board 
and for the annual tax levy that supports the district. 

Another source of feedback on program effectiveness is changes in enrollments. The 
school has learned that one of its strongest avenues of recruitment is referrals from students 
and graduates. In general, there always has been positive "word of mouth," but discontent 
among students is reflected quickly in lower enrollments. Students' satisfaction also is 
evident from their enrollment and attendance patterns. Students attend the school 
voluntarily, and they "vote with their feet" if they are not satisfied. 

Finally, students also make their concerns known directly to their instructors. They 
are open about their successes and frustrations, and they provide feedback on everything 
from examinations to placements to specific occupational competencies. 

Feedback Regarding Student Progress* Student performance on within-course 
measures and tests and their progress in mastering the occupational competencies provide 
the most direct feedback about their progress. Employers also provide information about 
student competence. Employers express their opinions most directly through their hiring 
decisions and their informal contacts with instructors when they are seeking potential new 
employees. In addition, employers who are involved in end-of-program student assessment 
make their judgments of the quality of student performance known directly to instructors 
and students. 

The State Department of Education also provides important feedback regarding 
employment status based on its semi-annxial follow-up surveys of students who complete the 
program. These reports include information on the employment and training status of 
program graduates, the relevance of their training to their employment, and the satisfaction 
of employers. 

Feedback Regarding Employer Satisfaction* The state follow-up survey provides 
some direct evidence of employer satisfaction with student graduates. However, much more 
useful and immediate feedback comes from employers themselves. Employers let instructors 
know directly if they are dissatisfied with the capabilities of students who complete the 
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progrram. Furthermore, the industry advisory committees provide a formal channel for 
employer opinions. MidAmerica has learned that the willingness of local industries to 
continue to participate in school-related activities is a strong expression of their judgments 
about the value of the programs. 

Change Mdchanisms 

There are both formal and informal mechanisms for change at the course, program, 
and school levels. These change processes are most easily recounted in terms of program- 
related decisionmaking. 

School-Level Changes. There are a number of formal change mechanisms that 
include the "teeth" of the accountability system. One is the formal biennial review of the 
school and each of its programs by the Office of Vocational Education. OVE makes general 
recommendations for changes at the school and program levels, and in the event of a serious 
shortcoming, OVE can rescind authorization for a program. Loss of authorization would 
translate into immediate suspension of the program, a dramatic change that has never 
occurred at MidAmerica. 

Less drastic actions are taken to improve or change programs long before they reach 
the stage of state decertification. The district superintendent conducts an annual internal 
review of each school and its programs. Schools are expected to describe program 
accomplishments, concerns, and strategies for improvement. 

Within the school, there is an administrative council consisting of one representative 
from each program and one representative from each special service area that meets 
quarterly to discuss schoolwide administrative issues and recommend changes. However, 
most change occurs at the program level. 

Program-Level Changes* In response to the district's annual review, the school 
principal requires each program to conduct its own annual review. To increaea the 
effectiveness of these reviews, the principal appoints a staff member from another program 
area to serve as a member of the program review team. As a part of the aimual review, each 
industry advisory committee conducts a formal review of competencies, curriculum, 
materials, and student outcomes. The advisory committee makes recommendations for the 
changes it feels are necessary. 

Coiirse-Level Changes. At the course level, instructors make ad hoc decisions about 
instructional strategies, course content, assessment methods, and course materials on an as- 
needed basis. 
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SUMMARY 

In this section we have described a notional, but we believe representative, vocational 
school. Our purpose has been to portray the information-rich nature of the environment in 
which vocational schools operate. At the same time, we hoped to point out that much of the 
information on which vocational schools operate is not easily captured in the summary 
figures and tables that are so often a mainstay of reporting requirements. For example, to 
say that each industry advisory committee met twice during the past year does not capture 
the flavor of involvement of local industry in program evaluation. Nonetheless, it is clear to 
us that local accoimtability systems; though often more informal than formal, exist widely in 
vocational education. 

In the next section, we discuss the practical limitations imposed on local accountability 
systems. 
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4. LIMITATIONS IN PRACTICE 



INTRODUCTION 

The MidAmerica example reflects a level of local accountability that is not always 
achieved in practice. In order to illustrate the features of the accountability model, the 
description of MidAmerica omitted any discussion of the shortcomings that often exist in 
local accoimtability networks. However, a discussion of local accountability is incomplete 
without an examination of practical limitations that can constrain accountability systems at 
the local level. In this section we turn our attention to these limitations. 

The model diagrammed in Figure 2.1 serves as a basis for discussing the shortcomings 
of local accountability systems. Most of the problems we have seen in practice can be traced 
back to deficiencies in goals, measures, feedback loops, organizational change mechanisms, 
or the interactions among these components. When one of th 9 components is missing or 
weak, it interferes with the effective functioning of the system. 

These four elements are highly interconnected, and each one afFediP the others. 
However, the clearest way to discuss their limitations is to consider the elements 
individually while holding the others constant. For example, in the discussion of measures, 
we assume the other elements in the accountability system are in place, i.e., the school has 
appropriate goals, there are mechanisms for communicating between administrators and 
constituents, and the school has procedures for adjusting the program in light of feedback 
that is received. If these conditions do not exist, any deficiencies in measures are 
exacerbated by problems with the other components, but the deficiencies in measures 
remain. Examining the components individually simplifies the discussion considerably while 
not limiting the generality of the conclusions we draw about measures or the other elements 
of local accoimtability. 

In the following paragraphs we will describe some of the common limitations that 
occur with each of the components of a local accountability system. Unless otherwise noted, 
the discussions of limitations are based on feedback from local program staff and 
constituents. All examples are derived directly from our site visits. As a rhetorical device, 
we will describe idealized components first and then analyze ways in which actual systems 
fail to achieve these ideals. 

There are some constraints we will not address, including external pressures from the 
state and federal levels and unusual local conditions that may not generalize. These 
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influences can take many forms. For example, state policymakers often establish goals that 
affect all vocational programs; federal legislation currently mandates the adoption of 
statewide systems of measures that will be required of all local programs; state funding 
formulas affect local administrators' options to change local programs, particularly their 
ability to initiate new programs in response to local demand. In Figure 2.1, these factors are 
represented in the area labeled state and federal, and we treat them as exogenous elements 
in the local accoimtability model. We will discuss influences from state and federal sources 
briefly in our conclusions and at greater length in a subsequent report. 

Finally, there are other local factors not included explicitly in the model that can affect 
accoimtability systems. These include such things as collective bargaining agreements and 
local political pressures. Such factors do affect administrators and instructors, particularly 
in their role in the change process. However, these elements are too idiosyncratic and 
dependent on local context to be included in a general model. To the extent that we have 
specific comments to make about such factors, they will be included in the discussions of the 
four accoimtability elements. 

GOALS 

The Role of Goals 

Educational systems are accountable to many constituencies: students, parents, local 
businesses, and local, state, and federal agencies, and each of these constituencies has a set 
of implicit goals relating to vocational training and employment. Students and parents are 
likely to share the goals of obtaining affordable education and training that lead to 
employment with opportunities for lifetime advancement. Local businesses desire a qualified 
labor pool and a free or inexpensive source of additional training. Local governmental 
agencies focus on the well-being of the local economy that they often translate into low 
unemployment levels. State agencies desire the same thing but balanced across the state. 
They also may be interested in promoting an equitable sharing of education and training 
resources across the state. Federal agencies, while interested in providing for a growing 
nationwide economy, often have as an equal goal equity in the provision of services to 
minorities, the handicapped, and underprivileged groups. 

These goals are communicated, formally and informally, to school district and program 
administrators, and they represent the de facto standards against which the performance, 
progress, or success of the local education and training system is judged. At the school and 
program level, goals arise from and are an attempt to crystallize the interests and desires of 
local, state, and federal constituencies. Thus, federal goals tend to be more broadly conceived 
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and stated than state and local goals, which in turn find expression in specific school-level 
goals. 

In addition to the felt need that often drives school systems to develop their own 
formal goals, federal legislation, i.e., the Perkins Act, requires states to have explicit goals for 
vocational education; a clause in the act says that states must 

develop and implement a statewide system of core standards [emphasis added] 
and measures of performance for secondary and postsecondary vocational 
education programs.^ 

It is clear from our discussion above that goals can be arranged into a natural 
hierarchy, from the relatively broad: 

By the year 2000, every adult American will be literate and will possess the 
knowledge and skills necessary to compete in a global economy and exercise the 
rights and responsibilities of citizenship.^ 

to the very narrow: 

Accoimting clerks will be able to "classify and record transactions, assets, 
Uabilities, capital, revenue, and expenses.**^ 

However, it is not always the case that only broader goals are generated at the federal 
level and narrower goals are generated at the local level. For example, the Perkins Act 
contains some relatively narrow goals, such as its requirement that states develop a system 
of measures and standards, and schools often adopt relatively broad goals, such as "providing 
opportunities for personal growth through basic vocational education and upgrading the 
knowledge and skills needed to keep pace with changing te-iinology." Ideally, each higher- 
level goal becomes translated into an increasingly larger number of more locally specific 
goals as we move from national to state to local to school to individual goals. In any case, the 
final result should be a set of interlocking goals. In addition to translating higher-level goals 
into lower-level goals, additional goals of regional or local interest may be added by each 
succeeding lower organizational level. 



^SEC. 115. STATE AND LOCAL STANDARDS AND MEASURES. While we recognize that 
goals and standards are not necessarily interchangeable — certainly not when standards represent a 
minimum acceptable level of performance — ^we believe that the standards required by the act are, in 
effect, goals for states to strive to achieve. 

^The National Council on Education Standards and Testing, Raising Standards for American 
EducatioTU A Report to Congress, the Secretary of Education, the National Education Goals Panel, and 
the American People, Washington, D.C., January 1992. 

^Taken from materials provided to us during a school system site visit. 
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The research literature on goal setting indicates that for goals to appropriately affect 
behavior, each goal must be seen as meaningful and realistic, each must be subscribed to by 
all interested parties, and each must be stated in such a way that it is possible to know if or 
when it has been achieved. It is also perhaps obvious but important that goals not be 
conflicting. With multiple organizational levels, this is often difficult to determine. 

Thus, an ideal system of goals is one that is broad at the highest level and interlocks 
with and becomes increasingly specific at lower levels/ Constituents at each level must 
imderstand and support the goals of the organizational level above them and develop goals at 
their level that support higher-level goals. As much as possible, goals must not conflict and 
must be stated in a way that allows constituents to measure progress toward them or 
achievement of them. Finally, if a set of goals is not equally important (and they usually are 
not), then a set of priorities must be established, promulgated, and supported. As often as 
not, a m^or stumbling block is not that the goals themselves conflict but that limited 
resources do not allow all goals to receive equal support. In this case, what may be perceived 
as conflict among the goals is really conflict between groups that have different goal 
priorities. 

Limitations in Goals 

As we have already suggested, the major shortcomings that we find in goal systems 
are as follows: 

• Higher-level goals that are not supported by interlocking goals at the action 
Gocal) level; 

• Goals that are sxifficiently vague or broad so that it would be impossible to know 
when or if they have been achieved; 

• Failure to obtain xmderstanding of and support for the goals from relevant 
constituencies; and 

• Unstated or xmclear priorities among goals. 

*In their discussion of a system of standards and assessments, the National Council on 
Education Standards and Testing, op. cit., provides a worthwhile scheme for oi^anizing standards that 
is readily adaptable for organizing local goals. They suggest the following components: (1) an 
overarching statement that provides a guiding vision, (2) content standards describing knowledge, 
skills, etc., to be taught, (3) student performance standards, (4) school delivery standards, i.e., capacity 
and performance of a school, and (5) system performance standards for each higher administrative 
level, e,g., district, region, state, etc. 
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Constituents' goals are ultimately expressed in mission and goal statements at the 
school and program levels. To illustrate some of their shortcomings, we turn to specific 
examples of goal statements. Figure 4.1 shows a sample mission statement from cne of the 
schools we visited. The formal school-level goals of this same institution are 



The Board of Education, Administration, and Staff of (name) Area Vocational- 
Technical School are committed to providing vocational skills training for 
individuals and for new and existing business and industry of the district. In 
order to provide this training, the school will offer programs, courses and 
seminars. The goal of each will be that the student gains vocational and 
technical entry level, upgraded, or specialized job skills. 

Figure 4.1 — Sample School Mission Statement 

listed in Figure 4.2. We could also provide a complete list of goals at the program and course 
levels, but these lists are prohibitively long and too detailed to include here. At the program 
and course levels, goals frequently appear as statements of required occupation-specific task 
proficiency (e.g., for an accounting clerk "journalizing transactions into multicolumn joumaD 
and employability skills (e.g., demonstrate self-control). 

The mission statement represents the overarching goal for the school and is 
presimaably embodied in the goals listed in Figure 4.2. The goals are indeed lofty and 
admirable. Unfortunately, such goal statements are often developed, occasionally revised, 
and then consulted only once or twice a year. For example^ though we do not know how 
frequently these goals are reviewed, this particular set of goals came with a notation that it 
had been adopted in October 1973 and revised in November 1978, December 1986, and July 
1990, i.e., only four times in 17 years. 

Two things are obvious from the list of school goals in Figure 4.2. First, none of the 
goals is stated in a way that permits the school administration to know definitively whether 
it was met. For example, what does it mean to assist a student in determining individual 
vocational goals? And how would we know we had done it? Would a reference section in the 
school library on career information be sufficient? Would every student have to be assisted 
for this goal to be met? When specific levels of goal attainment are left unspecified, it is 
tempting to consider the goal to be an absolute or to accept any degree of success as complete 
success. 

Second, because these goals are open-ended, it is clear that there are not sufficient 
resources available to meet every one of them— perhaps not even sufficient resources to 
address every one of them. Yet there is no clear priority of goals. Knowing how or when to 
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The committment of the Staff, Administration, and Board Members is centered around 
the needs of tlie students and the organizations we serve. 

We are committed to: 

Assisting the student in determining individual vocational goals. 

Assessing the individual's basic skills and providing prescribed instruction. 

Providing services which address the needs of special populations. 

Maintaining a spirit of service which allows the school to respond to the needs of 
individual students. 

Assisting local organizations in achieving their goals through customized training for 
their workers in the job skills necessary for their workplace. 

Cooperating with all local and state agencies or groups involved in developing growth in 
jobs and productivity. 

The basic responsibilities assigned us by the (state) Constitution to make available 
quality vocational and technical education to all the citizens jf our district. 

The continual evaluation of the effectiveness of o\ir instructional program and to the skill 
level of our student product in the world of work. 

A continually evolving curriculum in order to keep pace with the workplace. 

The use of research and advisory committees from business, industry, the professions, 
and labor in developing the courses of study that will make up our school's programs. 

Assisting all students enrolled at (school) in developing skills of citizenship and 
leadership. 

Providing opportunities for personal growth through basic vocational education and 
upgrading the knowledge and skills needed to keep pace with changing technology. 

Cooperating with all other branches of (state)'s educational system when it will provide 
opportunities for students or graduates or where it will provide for a better use of 
resources. 

Figure 4.2— Sample School Goals 

make trade-offs among goals is farther complicated by the lack of clear statements that 
would set a target for each goal. Would the system reduce the assessment of basic skills to 
fund a different level of service to special populations? 

Because of the lack of clear operational definitions in these goals, this school will find 
it difficult to judge progress and will be frustrated by conflicting implicit priorities among the 
staff and in the community. On the other hand, the administration will always be able to say 
positive things about what the school is doing (if you do not have explicit goals, it is easy to 
say that you are achieving them or "working toward them"). 
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This is not to say that broad goals are inherently "bad." Clearly, they provide a 
general direction within which programs can function. However, to the degree that no 
shared understanding exists about their meanings or how to translate these meanings into 
actions, there may be conflict o^ friction in the system. Sometimes goals are left broad in an 
attempt to bridge underlying disagreemaits. But this is more useful as a negotiating tool 
than as an accountability vehicle. 

Of course, one can err in the opposite direction as well. Highly operationalized goals 
can also wreak havoc on a system. As we will point out in our discussion of measures, there 
is often a tendency to place more faith, and hence more importance, in those things that we 
can easily quantify, sometimes to the decrement of more important but less easily 
measureable goals. 

Even the best of goals can fail if they are not understood and supported by the staff 
and the administration. Several things can go wrong in promulgating goals throughout the 
system. For example, an instructor in one school that we visited was never apprised that his 
job included the responsibility for placing students. Two things happened as a result. First, 
the instructor did not know to focus effort on student placement, so placement suffered 
initially. Second, the instructor was given a poorer performance review than perhaps he 
deserved. In this case, the goal was simply not communicated properly to the person 
responsible for achieving it. It is often equally inappropriate to say that a given goal is 
everyone's responsibility— with no incentives or sanctions for individuals to work toward it- 
achievement. Such a goal quickly becomes no one's responsibility. 

Goals often succeed or fail because of the support that is provided by administrators. 
While we may be tempted to think of support as simply providing the means to accomplish a 
task, it also includes the development and maintenance of an atmosphere that rewards task 
performance. Consider a goal to use advisory committees to ensure that curricula remain up- 
to-date and relevant to community needs. A school can provide support for this goal by 
providing time and space for meetings to occur. If there iz little incentive to hold the 
meetings, all the time and space in the world will not achieve the goal. The school needs to 
create an incentive for holding these meetings. One method might be to require that 
committee reports be part of an instructor's performance review package or that instructors 
be required to present committee reports twice a year at a department faculty meeting. The 
important point is tiiat supporting a goal means providing the time and resources so that it 
can happen and the incentive structure that will encourage it to happen. 

This leads to our final point with regard to goals: the importance of clearly stated and 
supported priorities among goals. In the absence of explicit schoolwide priorities, individuals 
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will develop and act on implicit priorities. These implicit priorities are likely to be a source of 
conflict in a school. For example, many vocational schools offer customized training for local 
employers. Unless there is a clear statement of the priorities for these activities vis-a-vis the 
school's regular program of classes, departments may enter into conflicts over space and 
instructor time. Limited time and resoin'ces and an abundance of goals demand that 
priorities be set. Priorities guide individuals in allocating time aiid resources among 
competing demands, as in the example above. 

Goals are crudai to an accountability system. However, no matter how clearly stated 
or strongly supported, if a system is imable to measure progress toward achieving them, 
accountability will fail. In the next subsection we discuss the importance of me^isures and 
ways in which fail\ires in the measurement system can affect local school accoimtability 
systems. 

MEASURES 

The Role of Measures 

Although the term "measure" may seem abstract or theoretical, the concept of a 
measure is familiar to everyone involved in education. A measure is nothing more than a 
quantitative index describing the status of a phenomenon. Test scores are a common 
educational measure, but educational measxires also include counts and tallies of outcomes 
(e.g., course enrollment, attendance, participation in extracurricular activities) and ratings of 
performance (e.g., grades, judgments about the adequacy of performance on job-related 
tasks). Common vocational education measures include scores on tests of occupational 
knowledge, tabulations of the percentage of graduates finding jobs in a particular 
occupational field, and proportions of occupational competencies mastered by students. 

Whether we like it or not, measures play important roles in our society. They provide 
quantitative information about diverse phenomena from athletic performance to judgments 
about beauty. Such data seem to hold a tremendor.s fascination for people and to wield a 
powerful influence over our lives. Things we can quantify seem to carry greater weight than 
things we cannot. 

This is true in vocational education as well. Although there are many valuable 
vocational outcomes that do not translate directly into simple measures, e.g., self-esteem, 
"quality of the workforce," and deportment, when people think about vocational program 
outcomes, they frequently think in quantifiable terms. Under the circumstances, the 
confusion between goals and measures alluded to in the previous subsection is 
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understandable. It is easy to focus on outcomes that can be counted while overlooking the 
importance (or lack of importance) of the things we are counting. 

The fascination with and respect for data pervade the current debate about 
educational reform. The campaign for educational choice relies upon measures of school 
quality to inform parental decisionmaking. Standardized test scores are one such measure 
that seems to have tremendous credibility both for parents and educational policymakers. 
The re-authorization of the federal vocational education act mandates the establishment of 
measures and standards for program evaluation and improvement purposes. Both of these 
reform efforts are geared toward increasing accountability in education, and they would be 
crippled without an appropriate set of measures. 

The chief role that measures play in accoimtability is to provide evidence of the 
attainment of goals, and the most important measures are those that are goal-related. 
Vocational programs frequently have goals for students that involve learning occupational 
knowledge, performing job-related skills, completing a sequence of courses leading to 
competency in an occupational area, and finding employment or pursuing additional 
schooling. Consequently, measures of occupational knowledge, job-related skills, course 
completion, and placement are highly relevant to vocational programs. 

What characteristics should measures have if they are to be used as tools for local 
accountability?^ The most important features of individual measures are consonance with 
goals, technical quality, and meaningfulness. In addition, the collection of measures used to 
describe a goal should be sufficient to portray overall status with respect to the goal This 
characteristic of sets of measures can be called sufficiency. Consonanpy and sufficiency are 
discussed in the next paragraph, followed by quality and meaningfulness. 

An individual measure is consonant with the goals of a particular school or program if 
it provides information that is relevant to an endorsed goal of the school or program. For 
example, the percentage of students who complete a wait on teamwork skills is a measure 
that is relevant to the goal of preparing students to be effective workers in the modem 
workplace. As a single measure, it is consonant with this broad goal. However, completion of 
the teamwork unit is inadequate if it is the only measure of this goal. Many other 
dimensions of preparedness would need to be assessed before one could judge attainment of 
this goal with confidence. In this case, a set of measures is needed to assess performance 
with respect to this particular goal. Other constructs one might want to measure include 
personal management skills, problem solving ability, communication, and basic skills. A set 

^This discussion is derived more from psychometric literature than from specific -jmments 
made by interviewees. However, all examples are based on incidents reported during our site visits. 
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of measures would be sufficient if it provided adequate evidence to judge attainment of the 
goal. 

The technical quality of a measure usually is judged in terms of reliability and 
validity. For the purposes of this nontechnical discussion, reliability can be equated with 
accuracy and validity with appropriateness. A measure is reliable/accurate if it produces a 
score with a minimum of error. Measurement errors can come from many sources: 
ambiguous directions, poorly written questions, and Tbuman errors'* in compiling information 
or computing statistics. In tests, scores can be affected by the celection of questions or the 
choice of question formats. One way to determine the reliability of a test is to administer it 
two or more times to the same individuals. This process will seldom produce identical scores; 
however, if the tests are reliable, the scores will be qxiite similar. Large differences in scores 
are an indication of unreliable tests. To be useful as indicators of goal attainment, measures 
must be accurate. 

A measure is valid/appropriate for a particular purpose if it reflects the condition it is 
being used to represent. In the example above, the passing rate on the teamworl. unit might 
not be a valid indicator of workplace preparation. One person could pass the unit and be a 
poor employee; another could fail the unit and be an excellent employee (because he or she 
did not take it seriously or because the imit did not address teamwork in the workplace well 
enough). In reality, appropriateness/validity is not a characteristic of the measure itself but 
of how it is interpreted by the user. To be effective for accountability purposes, measures 
must be valid indicators of the status of goals they are taken to represent. 

Finally, measures should be simple, clear, and direct enough to be understandable to 
constituents. This is what we mean by the broader term meaningful. Measures must be 
neither too complex nor too elaborate if they are to make sense to the average person. 

Limitations of Measures 

Measures are ineffective as elements of a local accoimtability system if they are 

• Not consonant with goals; 

• Of inadequate technical quality; or 

• Not meaningful to constituents. 

Measures that are not consonant with goals fail to provide necessary information for 
program improvement. Measures diat are imreliable or invahd give false signals about the 
status of the system. Measures that are not meaningful to constituents cannot be translated 
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into appropriate feedback. These t3T5es of deficiencies limit the value of the measures as 
tools for accoimtability. The following examples illustrate the practical limitations on 
measures we have encountered in vocational programs. 

Lack of Correspondence Between Measures and Goals* A set of measures fails 
to correspond to program goals if there are goals that are immeasured or measured 
incompletely. In this situation, constituents lack objective information to judge program 
success. For example, if the broadly stated goal of a cosmetology program is to prepare 
students to be successful cosmetologists, then students' grades in cosmetology courses 
provide some measure of the attainment of this goal. However, grades are an incomplete 
measure of this goal. They do not indicate specific knowledge of key elements of cosmetology, 
they do not differentiate between knowledge of facts and the ability to perform the tasks 
associated with the job, and they do not necessarily correspond to likely success as a 
cosmetologist 

In comparison, it might be possible to combine a larger set of measures to judge the 
attainment of the cosmetology program goal. Such a set could consist of the following 
measures: 

• A test of students' mastery of specific occupational knowledge; 

• Observations of students* performance on hair care procedures; 

• Measures of students* deportment and employability skills; 

• Students* grades in a planned sequence of courses leading to a certificate; and 

• Students* scores on state licensing examinations. 

In this instance, no single measure would provide adequate data to judge the 
program's success in meeting its goal. However, in combmation, these measures might be 
adequate to assess goal attainment.^ 

When goals are immeasured or measured incompletely, people have to rely on 
subjective judgments about goal attainment or they have to ignore the unmeasured goals. If 
concerned constituents have only their own subjective impressions to use as a basis for 
feedback to programs, decisionmaking can more easily become politicized. Similarly, if no 

collection of measures is sufficient only if it provides enough information to determine 
whether students are adequately prepared to be cosmetologists. This question could be answered 
empirically by comparing performance on the measiires with performance on the job. Such a 
comparison could establish the predictive validity of the measures. In this case, the measures may be 
madequate because they do not contain any indicator of social skills, which are likely to be highly 
correlated with success in this particular occupation. 
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data are collected to determine whether the program is meeting a goal (e.g., serving students 
with special needs), less attention is Jikely to be paid to this goal (e.g., the needs of these 
students). 

It is also possible to include measures that do not correspond to any goals. One must 
be cautious that such measures do not supplant goals and become the focus of 
decisionmaking. There is a natural tendency to attend to whatever da'-a are produced and, 
by extension, the implicit goals they instantiate. Recent emphasis op competency testing 
may provide a case in point; programs may attend to test scores and the actions that can be 
taken to raise scores while not attending to the original goal that scores were supposed to 
reflect, e.g., job preparation. More generally, the mere existence of data is a powerful magnet 
to attention, and collecting measures that do not correspond to explicit goals can raise the 
implicit goals embodied in those measures to a prominence they do not deserve. 

Measures That Are Technically Inadequate. There are a number of ways that 
measures can be inadequate in terms of quality. In technical, psychometric language, we 
define quality in terms of reliability and validity. Measures are reliable if they are accurate 
and consistent. For example, in most schools the registrar's report of the percentage of 
students who complete tlie cosmetology sequence in the allowed amoimt of time is likely to be 
accurate, and, if conducted again, it would likely yield the same results. In contrast, a follow- 
up survey of the percentage of completers who foimd jobs in a field relatc^d to cosmetology 
may not be as trustworthy. Such surveys usually have high nonresponse rates, and those 
who do respond may not be representative of all program participants. As a result, the 
results of the survey might change if it were conducted again and the conclusions drawn from 
the survey are likely to be inaccurate. Because of the importance of employment as an 
outcome of vocational education and the fact that it is frequently assessed through a survey 
of some kind, it is especially important for participants in a local accountability system to be 
aware of these potential problems. 

Measures are valid for a particular purpose if the inferences drawn from them (e.g., 
about the occupational knowledge of students and their preparation for work) are correct. 
For example, performance on the state licensing examination probably is a reasonable 
indication of readiness to be a cosmetologist. On the other hand, performance on a ten-item 
test developed by a commercial publisher in one state may or may not be an appropriate way 
to judge preparation for cosmetology in another state. 
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Rather than engage in a detailed theoretical discussion of sources of measurement 
error and threats to validity,'' we will describe a few examples of measurement problems 
associated with technical quality that are likely to be encountered in the vocational context. 
This is not meant to be an exhaustive review of potential measurement problems but an 
illustrative presentation of the ways in which data can be deficient in practice. These 
examples include questions about the technical quality of 

• Occupational competency testing; 

• Follow-up surveys of students who complete programs; and 

• Labor market demand projections. 

Occupational competency testing plays an increasing role in vocational education 
programs. A growing number of states and organizations are developing occupational 
competency assessments, and a growing proportion of these assessments are expanding 
beyond the use of paper-and-pendl multiple-choice tests to include alternative assessment 
techniques such as performance-based measures and portfolios. There are many ways tests 
can be unreliable or invalid for local accountability purposes. As the use of tests increases, 
these concerns grow. 

A brief review of reports about vocational testing programs currently available or 
under development suggested a number of potential psychometric problems, particularly 
with new forms of assessment. Alternative forms of tests generated repeatedly from item 
banks may not be comparable in difficulty, so scores may not be reliable. Short competency 
tests may not sample adequately from the domain of skills needed to perform a job, so results 
may fail to reveal significant deficiencies in students, yielding invalid impressions of 
competence. Measures of performance that pose hands-on tasks may have imprecise 
guidelines for scoring, so standards will vary from teacher to teacher. Collections of student 
work products in portfolios may reflect optimum performance under conditions in which 
outside assistance and revisions are permitted, not typical performance under joblike 
conditions. All these are potential threats to the reliability and/or validity of measures, and 
all may lead to conclusions about program performance that are incorrect. 

Measurement quality also is a concern in the case of follow-up placement and 
employment surveys. These measures often suffer from poor data collection procedures and 

''For such a discussion, see any comprehensive text on educational and psychological 
measurement, e.g., W. Mehrens and I. Lrehmann, Measurement and Evaluation in Education and 
Psychology, 4th edition, Holt, Rinehart and Winston, Fort Worth, TX, 1991. 
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low response rates. For example, in cases where follow-up data are collected by the State 
Department of Education or other state agency at some distance from the program site, local 
programs often find it necessary to supplement state eflforts with locally collected data that 
more closely match actual placements. This problem is lessened when state data collection is 
based on electronic linking of school records and employment data (e.g., unemployment 
insurance fund contributions), but this is done only in a few states. 

More often schools are responsible for generating their own placement data but are not 
given adequate resources to do a thorough job. When individual programs are given the 
responsibility of tracking their own graduates, conditions are ripe for errors. Programs tend 
to interpret responses in the most positive light. For example, they are likely to trust one 
student's report of a second student's job status or to accept a student's comments about job 
intentions in lieu of data on actual job placement. 

Projections of labor market demand are another area in which measures often are 
invalid. Many local programs receive projections from state or federal agencies to use to 
estimate local demand for training. This information should help programs better plan to 
meet the needs of local employers. However, aggregated demand projections often yield 
invalid estimates of local demand, and they can lead to poor program planning. For example, 
a statewide shortage of nurses may not translate into a local shortage. Similarly, locally 
generated data on employment demand are not always trustworthy. In both cases, 
employers have been guilty of responding in terms of the employees they would like to have 
under ideal conditions, not the employees they will actually hire. 

A related concern has to do with the robustness of measures in high-stakes contexts. 
There is ample evidence that test scores and other measures can be corrupted (i.e., scores no 
longer reflect underlying ability) as the importance attached to the measure increases.^ In 
the vocational context, this means that when the stakes attached to scores increase (for 
example, if it is necessary to pass a test to receive a certificate of completion in a vocational 
program), scores tend to rise irrespective of changes in actual knowledge. This occurs for 
many reasons, including familiarity with the tests after repeated use and conscious *1^ching 
to the test." 

Unfortunately, most measures are susceptible to corruption. Despite the rhetoric that 
people are designing a new generation of tests designed "to be taught to," many vocational 
competency tests are likely to be corrupted if pressures on scores grow. Fortimately, local 

®D. Koretz, R. Linn, S. Dunbar, and L. Shepard, 'The Effects of High Stakes Testing on 
Achievement: Preliminary Findings About Generalization Across Tests," paper presented at the annual 
meeting of the American Educational Research Association, Chicago, IL, April 1991. 



accountability concerns do not usually create such high stakes for performance, but they can, 
and certainly statewide influences increase the importance of performance and place added 
demands on measures. 

Measures also may be invalid as accountability tools when external conditions, such as 
unemployment rates, affect them. Program performance, particularly placement of 
graduates, is not solely a function of training effectiveness; it also is affected by local 
economic conditions. For example, placements may decline in a recession though the quality 
of the training and the skill of the graduates have not changed. Under these circumstances 
it would be incorrect to use a measure of placements to indicate attainment of specific 
training goals. The measure would still be a valid reflection of community demand, but it 
would not be a fair indication of graduate skill or instructional quality. Consequently, one 
must be cautious about interpreting outcome measures that are linked to local economic 
conditions. 

Measures That Are Not MeaningfuL The last criterion for effective measures is 
meaningfulness. There are many ways in which measures may fail the test of 
meaningfulness in practice. Measures are of Hmited value if they are unclear or confusing 
(e.g., they are statistically complex), if they are not available in a timely manner, or if they do 
not address questions that are important to constituents. 

Coxmts, tallies, and percentages reported at the student or program level are generally 
well imderstood, but not all measures are this clear. Complicated learning-style profiles or 
scaled results from locally constructed occupational competency tests may be too complex or 
obscm*e to be imderstood easily. Some of the worst problems occur when programs make 
"statistical adjustments." For example, one college reported that "132%, or 632 of the 477 
students scheduled to graduate from a real estate training program, went on to graduate.**® 
This creates an impression that the program was phenomenally successful (or ridiculously 
incompetent). In fact, the school had no data at all on the number of students who foimd jobs 
in real estate. It may well have been true that this measure was reliable, but it was hardly 
meaningful to constituents. 

Timeliness is an important attribute of measures because most school-related 
decisions are time-dependent. Students and parents have to make enrollment choices by a 
particular deadline, teachers and industry advisors have to make curriculimi choices prior to 

was explained as follows: The number of students scheduled to graduate in 1988—89 was 
defined as 80 percent of the number of students enrolled in 1987-88. Graduates were defined as the 
nxmiber of students listed as Completers on the NCES 2404-A Postsecondary Enrollment and 
Completion Report. 
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the start of the term, and administrators have to make hiring decisions on an annual basis. 
For goal attainment data to be useful, they need to be available in a timely manner. 

Finally, measures should provide data that are responsive to the kinds of questions 
constituents are likely to ask. There may be many ways to gather information relevant to a 
particular goal, e.g., to serve students with special i^eeds. Some measures will be more 
relevant to employers, instructors, parents, or students than others. When choices exist, 
schools should opt for measures that have the greatest meaning for their multiple 
constituencies. To be meaningful, measures must be imderstandable, timely, and responsive. 

In conclusion, there are many ways in which measures can fail to fulfill their role in an 
accoimtability system in practice. One must not assume that because something is 
quantitative it is good. Nor should the reader be left with the impression that a measure is 
poor if it is qualitative. In both cases the important questions to ask are whether the data 
are consonant with goals, reliable and valid, and meaningful to constituents. These are the 
criteria that should be used to judge the quality of measures in a local accountability system. 

FEEDBACK 

The Ro!e of Feedback 

We use the term "feedback** to refer to two related but distinct processes: (1) the flow 
of information about school-based programs, objectives, and outcomes to constituents and 
school staff that forms the basis for judgment about how well the total vocational education 
system is working, and (2) the flow of information (usually judgments, opinions, and 
interpretations) between the various groups of constituents and the school administrators 
and staff (see Figure 2.1). 

So defined, feedback is a continuing process that takes many forms — both formal and 
informal, direct and indirect. For example, course enrollment levels are one piece of 
information about the health of a given program. This information is published formally in 
an annual report, and it is also available informally to instructors and staff throughout the 
year based on their direct observation of classes. 

The level of employer satisfaction is another indication of program performance. 
Satisfaction can be measured directly through surveys and conversations, or it can be 
inferred indirectly from employers* participation in advisory groups and employers* continued 
interest in hiring program graduates. In this manner, constituents* actions constiti\te 
indirect feedback about their judgments and opinions. This is true for students and parents 
as well as for employers. 
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Limitations in Feedback 

Feedback includes many forms of information sharing, and it is subject to all the 
problems that can plague human commimication. This includes insufficient information and 
inaccurate or insensitive commimication. In the vocational education context, some feedback 
occurs within organizational boundaries, while other potentially more important 
communication with constituents must travel across organizational boimdaries. This may 
add an additional layer of difficulty if the organizational culture and the "outside" culture 
have different standards or expectations regarding commimication. For the purposes of this 
discussion, limitations in feedback will be categorized as 

• Insufficient communication; 

* Inaccurate commimication; and 

♦ Low signal-to-noise ratio — a high rate of communication with very little useful 
content. 

Insufficient Communication. Information is one key to effective action. For 
program administrators and staff to initiate, modify, or discontinue programs rationally (see 
section on organizational change mechanisms), they must have valid, reliable, and 
meaningful program information to guide them. For example, to the degree that instructors 
are isolated from local employers and do not receive feedback about employers' hiring 
priorities, instructors will be unable to adjust program content to employers' changing needs. 
Similarly, to the degree that school administrators do not receive job placement information 
on program graduates, their decisions regarding program expansion or contraction will 
suffer. To the degree that community members lack information to judge the value of their 
local vocational education and training system, they will be unable to shape it to their needs. 
They may be less willing to provide the fiscal support it needs as a result. 

Communication may be insufficient for several reasons. People tend to err on the side 
of sharing too little information rather than too much. Those who have information to 
communicate often feel that they have communicated more than those receiving information 
feel they have been given. One of the most difficult aspects of communication is judging the 
appropriate amount of information to share. 

Second, a person may limit the amoimt of information he or she communicates 
publicly for political reasons since the control of information contributes to the exercise of 
power. Those who possess information have an advantage over those who do not. Sometimes 
the conscious restriction of information is quite subtle. For example, business 
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representatives who join together in program advisory committees are by their nature local 
competitors — e.g., beauty salon operators in the same city. This can generate considerable 
pressure on committee members to be less than entirely forthcoming with information. In 
one location, an advisory committee felt the need to have a formal written agreement 
concerning the use of information obtained through committee deliberations. The members 
believed that this agreement allowed a much freer exchange of information. 

Inadequate communication also can arise as the result of ineffective organizational 
arrangements. For example, one community college we visited had a centralized placement 
office that carried out all of the placement support functions for program graduates. While 
this specialization appeared to be an efficient use of resources, it created an unanticipated 
buffer between instructors and loqal employers. Because instructors were not responsible for 
job placement, they failed to receive the natural flow of communication about employers' 
needs and program content that occurs during the placement process. 

Lack of Accuracy of Communication. Information can be conmiunicated 
inaccurately or insensitively for many reasons. First, human communication is susceptible 
to many forms of bias, both intentional and xmintentional. Intentional bias occurs when 
someone purposely distorts reality. Unintentional bias is more difficult to detect and may be 
*ar more common. A local employer who is especially happy or imhappy with the quality of a 
program graduate can provide unintentionally biased information. If the employer is 
persuasive, a single strategically placed remark can have a dramatic effect on others' 
perceptions of the program — regardless of whether the remark truly reflects the overall 
quality of program graduates. The employer may have no intention other than to report his 
or her experience with a single graduate, yet a positive or negative anecdote can have much 
greater impact than a report filled with statistics. 

Second, information can be distorted to serve a particular agenda. Individuals can put 
their own "spin" on information by highlighting or downplaying either the positive or 
negative aspect of the information. For example, in one location we visited, a child care 
worker program was being discontinued. Although job placement rates were high, the 
program served primarily women, and the graduates were earning a relatively low wage. 
The administration decided to dismantle the program because, in their opinion, it 
perpetuated women in low-paying jobs. Clearly wage levels and placement rates provided 
two different perspectives on the success of the program. Administrators decided to 
emphasize one piece of information over the other; students might have put a very different 
spin on the information. We do not cite this as an example of poor or good judgment on the 
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part of these administrators; rather it shov/s how one piece of information may be 
emphasized xmwarrantedly over another. 

Low Signal-to-Noise Ratio. In some instances, there may appear to be a substantial 
flow of communication, yet very little usefiil information is being exchanged. The classic 
example of this is a political speech, but we do not find large amoxmts of noise and small 
amounts of content only in political rhetoric. Vast amoxmts of noise can masquerade as 
information in many other settings. In vocational education, this may take the form of 
undigested, unsummarized, xmsynthesized, or unanalyzed statistical information. For 
example, schools may publish page after page of course enrollment figures. If this 
information is not summarized or if additional contextual information is not present (such as 
trends in enrollment over time or local employment figures), the information is effectively 
noise that the reader must sort through. This is not to say that statistical reports are 
worthless; rather that it can be difficult to find the key information amid the noise. 

Why does this happen? Oft«n individuals feel that all information that has been 
collected should be distributed. Furthermore, it takes an experienced data analyst to find 
appropriate ways to summarize raw data without biasing the information. One method that 
can be used to ameliorate this problem is to provide summaiy information in the body of a 
report or presentation and to include the raw data in an appendix. Providing only the 
summary or only the raw data is less likely to be satisfactory. 

Communications with low signal-to-noise ratios have predictable effects. First,, 
individuals simply cease to pay attention to the information they are given. Decisions 
continue to be made but without the benefit of useful information. Second, increased time is 
devoted to sorting through the data to find and interpret the useful information that is 
contained amid the noise. Third, the noise is confused with iiseful information, leading to 
inappropriate conclusions and actions. 

To summarize, feedback represents the flow of information conveyed by the measures 
to administrators, program taff, and school system constituents, and the flow of information 
between administrators, staff, and constituents. Problems in feedback result in inaccuracies 
or distortion of information that can lead to poor decisionmaking. 

ORGANIZATIONAL CHANGE MECHANISMS 
The Role of Organizational Change Mechanisms 

Studies of organizational behavior have documented the tendency for organizations to 
persist in the same actions rather than to change.^^ Persistence occurs, in part, because 



^^J. G. March and H. A. Simon, Organizations, John Wiley & Sons, New York, 1964. 
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there are costs (both real and psychological) associated with change. Schools demonstrate 
the same inertia and the same tendency to maintain the status quo rather than change. The 
loosely coupled" nature of education, in v/hich hierarchical authority is lessened, may make 
schools somewhat more responsive to environmental pressures than other organizations,^^ 
However, this same ^'decoupling" may reduce the likelihood of large-scale change while 
increasing the ease with which individual classroom instruction responds to pressures for 
change. ^2 

Our observations of vocational education programs are consistent with these 
viewpoints. We found that large programmatic changes were infrequent. For example, it 
was rare for a school to initiate a new program or terminate an old one. On the other hand, 
change was common within specific occupational programs. Small modifications to courses, 
cxirriculum, equipment, student requirements, and assessment methods occurred regularly. 

In discussing organizational change mechanisms, we include both types of change, 
those affecting programs as a whole and those affecting elements within programs or 
services. At the school level, the director, principal, or e^iecutive committee is the chief agent 
of change; at the program level, the program coordinator, department head, or curriculum 
committee acts in this role. In small programs (which were quite common in the schools we 
visited) there may be only one or two instructors who are responsible for curriculum change. 
In abstract terms the role of organizational change in the accoimtability network : the same 
in all these situations. 

When a local accoimtability system is functioning effectively, change is fostered 
because constituents have power to reward or sanction the behavior of the institution. Not 
only can they make their opinions known regarding the success of the institution, they can 
act on their beliefs. Students can leave programs and share negative reviews with peers if 
they do not believe the school is achieving the goals that are most important to them. 
Likewise, businesses can withhold their participation — on industry advisory committees and 
as employers of program graduates— if they do not believe the school is achieving goals that 
are important to them. Similarly, instructors, parents, and commimity members have power 
as voters, board members, participants, and advisors to create incentives for achieving 
appropriate goals. 

G. March and J. P. Olsen, Ambiguity and Choice in Organizations . Universitetsforlaffet 
Bergen, 1976. ^ ' 

^J. W. Meyer and B. Rowan, •'The Structure of Educational Organizations," in J. W. Meyer and 
W. R. Scott (eds.). Organizational Environments , Sage, Beverly Hills, 1983. 
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Administrators have options for responding to constituent feedback (as well as their 
own evaluations of program performance), and they play the key role in change. In our 
model they are the recipients of information on system performance (measures) and of 
feedback from constituents on their interpretation of measures and the degree to which the 
program is meeting their needs. Administrators use these inputs to make decisions about 
continuation or modification of programs. 

Organizational change need not occur through a regular, formal, systematic process 
for a local accountability system to function. That organizations tend to change in 
evolutionary and episodic ways does not detract from the accountability model we described. 
Measures of program performance and feedback from constituents influence program 
decisionmakers, although it may take time for these effects to be translated into action. 

What are the characteristics of an ideal organizational change mechanism? Three 
elements seem crucial to us. First, the organization must be responsive. When data suggest 
that goals are not being met or when feedback suggests that constituents are dissatisfied 
with performance, actions need to be taken to understand and improve the situation. The 
key word here is improvement It is not enough to mollify critics, adapt measures, or 
discount results. What is required is movement toward a more effective way of doing things. 

The second characteristic of an ef rective change mechanism is that it must be forward 
looking. Schools must be willing to look beyond short-term satisfaction to intermediate and 
long-term goals. Business cycles can affect short-term demand for employees, so schools may 
need to ignore short-term fluctuations in placements to be responsive to long-term industry 
demands. Schools must avoid eliminating programs that may be viable in the long nm. 

Finally, organizations must be fair. They must respond to the needs of all constituents 
and not give undue weight to feedback from some groups over others. The adage "the 
squeaky wheel gets the oiF is often true in educational settings. An effective change 
mechanism is one that balances the needs of constituents, respecting the desires of large and 
significant groups without dismissing the wishes of small groups. 

Limitations in Organizational Change Mechanisms 

Organizational change and reform can falter for many reasons. In our visits to 
vocational schools we saw examples of decisionmaking and program reform procedures that 
were far from ideal. Among the shortcomings were the following: 

• Regulations that limit options for change; 

• Insufficient resources; 
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• Overattention to the needs of one constituency (e.g., employer groups); 

• Giving priority to short-term demands over long-term trends; 

• Difficulties balancing competing goals and principles (e.g., eqxiity versus 
placement); 

• Lack of formal procedures for change; and 

• Ineffective leadership for reform. 

Often options for local change are constrained by state or federal regulations and 
funding guidelines. In one state, new program funding was available only in occupational 
areas where state labor market demand projections indicated growth. However, local school 
adnmiistrators did not believe these projections accurately predicted local demand. 
Nevertheless, schools could not receive state funds for new programs rmless the programs 
appeared on the state's approved list. Other kinds of regulations can limit administrators' 
options for reform. One vocational school had a two- to three-year waiting list for enrollment 
into its nursing program, but it was not allowed to start additional classes because of limits 
placed on it by the State Board of Nursing. 

One of the most common restrictions faced by all educational programs, not just 
vocational education, is limited funding. In the vocational context, resource constraints 
reduce schools' ability to respond to changes in demand. In many states, funds ior vocational 
educational programs have been "capped," and schools receive no additional resources or only 
partial funding when they add students or programs. 

Another factor that can reduce the effectiveness of change procedures is overattention 
to feedback from industry. As an example, bowling alley operators in one community made a 
strong case that training was needed to prepare mechanics to repair automatic pin-setting 
machines. The school did its best to conduct an objective survey of demand, which lent some 
support to industry claims. The industry advisory group was adamant that the program was 
needed, and they were willing to raise funds for the capital expenditures necessary to 
prepare the facilities. Despite its reservations, the school accepted the group's help to 
prepare the facilities and develop the curriculum. 

The program was offered, but enrollments were insufficient to sustain it. After some 
investigation the school learned that the bowling alley operators themselves were 
withholding information from employees who might enroll. The owners were unwilling to 
refer employees because they did not want to pay the higher wages that trained mechanics 
could command. The results of the employer survey were misleading because owners had 
indicated **the type of employees they wanted, but not the type of employees they were 
willing to hire." In retrospect, the school believed it was persuaded by owners' desires 
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without an adequate assessment of owners' comraitments. The school compHed with the 
wishes of the advisory committee, partially out of respect for the employers. Unfortunately, 
the space devoted to the bowling machine repair program could have been used more 
effectively for other programs. 

A related problem occurs when schools attend to short-term demand without 
consideration of long-term needs. For example, one community collage created a program to 
train pulmonary therapists based on employer surveys that projected a strong immediate 
need. However, the needs analysis did not estimate turnover and continuing demand in the 
field. The school soon found itself with a program that could no longer place graduates 
because all the positions had been filled. 

Managing change can be difficult when administrators have to balance competing 
demands or competing principles. For example, one area vocational school eliminated its 
child care worker program despite continuing demand because the program was training 
women for an occupation the school identified as a low-paid, traditionally female, and "dead- 
end job." Administrators judged this training program to be an inappropriate use of 
resources that might better serve to develop more promising training opportunities.^^ In this 
case the school gave priority to principles over demand, to broader career-oriented goals over 
short-term employment goals. 

Finally, although it is an extreme case, some institutions act as if they have no 
mechanism for change. While effective vocational schools regularly update and redesign 
facilities to meet the changing training needs of their local commxinities, other schools seem 
to have Httle or no capacity for self-improvement. For example, one high school in an urban 
area provided vocational programs as part of a larger regional training consortium. The 
school itself did little to broaden the range of courses allocated to it or to improve the quality 
of its classes or facihties. One reason for this seeming indifference was that vocational 
education had little prestige at the school compared to college preparatory academic 
education. Another reason was that the school had little power to affect the allocation of 
vocational courses. Either through neglect, bureaucratic inflexibility, or the absence of 
leadership, the school made almost no efforts to improve vocational programs or facilities. 
Although this example was striking, we h ave no reason to beheve it is typical of vocational 
programs. 

Overall, there are a number of ways in which practical constraints inhibit 
organizational change mechanisms. Even when goals are well articulated, measures well 



i^They had not yet identified those opportunities and developed appropriate training proCTams 
at the time of our visit. & r & " 
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defined, and feedback from constituents prevalent, administrators may be ineffective in 
translating these elements into action. Administrators are influenced by politics and by 
external factors beyond their control. They are limited by their ovm capacities as leaders, 
and their actions can be affected by weaknesses in their change strategies. This includes 
failure to be responsive, overattention to short-term solutions, and susceptibility to pressures 
from vocal groups. 

SUMMARY 

Theoretical models can be useful tools for understanding social phenomena, and the 
model we proposed in Figure 2.1 is helpful for describing accountability at the local level. 
However, theoretical models of social programs have limits — ^they describe conditions in ideal 
terms that are not necessarily implemented in specific situations. In this section we 
presented a number of practical limitations drawn primarily from our study sites that can 
reduce the applicability of our model and the effectiveness of local accountability systems. 
These limitations can be described as deficiencies in the components of the model — goals, 
measures, feedback loops, and change mechanisms — and the interactions among them. 
Greater familiarity with the functions of accountability in practice will improve our model 
and its usefulness as a policy tool. 
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5. SUMMARY AND CONCLUSIONS 



In this study we sought to examine accountability relationships in vocational 
education at the local level as a complement to earlier NCRVE-sponsored research on 
accountability at the federal level. ^ Our purposes were to investigate the nature of local 
accountability in vocational education and to examine the effectiveness of such local 
accountability systems. 

To this end we reviewed the literature on accountability in vocational cUucation, and 
we conducted extensive interviews with constituents of vocational programs in five states. 
Based on these data we developed a model of accoxintability at the local level and a collection 
of anecdotes about limitations of the model in practice. Both the model and the practical 
limitations should be useful in future research on accountability and on the effects of changes 
in federal and state policy regarding vocational education. 

For example, as a result of this study we believe that accountability systems are 
impaired if the components — goals, measures, feedback, and change mechanisms — are out of 
balance. That is, if they differ dramatically in terms of emphasis, credibility, sophistication, 
and efficacy. Recent federal efforts to promote accoimtability in vocational education have 
focused primarily on measurement and, to a somewhat lesser extent, on change. Much less 
attention has been given to goals and goal setting. One consequence of this emphasis on 
measurement is that measures may begin to supplant goal setting rather than having goals 
drive the choice of measures. To what extent is this occurring? What are the goals that are 
implicit in the measures being adopted, and how do they differ from the goals constituents 
hold for vocational education? These are empirical questions that might not be asked unless 
one has a systemic perspective on accountability. The general model of local accountability 
described in this study provides such a perspective, and it can help researchers generate 
appropriate questions. 

We draw four main conclusions from this study: 

• There are local accountability systems operating in vocational education 
programs that are distinct from stat^- or federally imposed standards and 
requirements for program review, 

^Hill, Harvey, and Praskac, 1992, 
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• These systems can be described reasonably well by a model with four 
components: goals, measures of goal attainment, feedback from constituents, and 
organizational change mechanisms. 

• The quality of these components and the relationships between them account for 
much of the variance ix» local accountability. 

• Practical constraints exist to limit the effectiveness of these systems. 

Understanding these limitations can lead to prescriptions for improving local accountability 
and to better understanding of the impact of state and federal policy. 

To elaborate on the first conclusion, we found it makes sense to talk about local 
accountability in vocational education as distinct from federally mandated accountability. 
This, in itself, is an important result, for many would argue that one of the problems facing 
general education is that schools are not accountable to students and parents. In vocational 
education the situation is different. The nature of the educational enterprise is such that one 
often finds meaningful local goals, clear signals of goal attainment, and additional local 
constituents with a vested interest in outcomes. All these create a framework for local 
accountability, and although this framework often is informal, it is systematic. 

Second, a simple model captures the important elements of this accountability system. 
Every school system we visited had formal goals. Although goals may differ for different 
stakeholders, if they are shared, they can be the basis for accountability. Next, there must 
be ways to measure and publicly communicate about attainment of goals. Most measures are 
defined in terms of outcomes, e.g., program completion, skills attained, and placements 
achieved. Feedback is the third component. Measures must be couimunicated to program 
staff and constituents, and the opinions and interpretations of constituents and staff must be 
communicated back to program administrators in some form-either directly through 
discussions or indirectly through actions. Finally, the system must have some mechanism 
for change in response to feedback. In most cases the responsibility falls on the shoulders of 
administrators, who may be more or less effective at translating feedback into program 
improvement 

Third, the quality of these four components affects the capacity of the local 
accountability system to produce program improvement. Poorly stated or unrecognized goals 
reduce the focus of administrators. Measures that are not consonant with goals limit 
judgment about program effectiveness. Lack of feedback from constituents fails to portray 
adequately' their needs. Ineffective change agents fail to use the information provided by the 
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system to make necessary changes. Limits in these components interfere with the 
effectiveness of the overall accomitability system. 

Fourth, it is possible to identify a n\unber of the most common practical constraints 
that limit the effectiveness of goals, measures, feedback, and change mechanisms. For 
example, goals are often stated in ways that make it impossible to know if they have been 
achieved. Such goals provide little basis for judging progress while permitting schools to 
draw positive conclusions about their effectiveness. Even when schools operationalize goals 
well, they seldom assign priorities that would help guide resource allocation decisions. 
Similarly, when measures are lacking, people rely on subjective judgments about goal 
attainment, which can overly politicize the decisionmaking process. The use of measures of 
imknown or inadequate technical quality may lead to unwise program planning and reform. 

Feedback can be reduced or distorted in ways that constrain its effectiveness. For 
example, actions that are taken to enhance services (providing job placement specialists) can 
have the opposite effect by limiting communication between instructors and employers, a 
critical source of data on student mastery and program effectiveness. Focusing on high job 
placement rates can mask the fact that the graduates are earning poor wages. Similarly, 
educational change mechanisms can be seized by well-meaning administrators with strong 
visions, whose actions may or may not farther the needs of their constituents. 

The identification of common limitations such as these in local accountability systems 
has at least two potential benefits: We can begin to identify criteria to use to evaluate local 
accountability systems. In addition, we also can develop prescriptions for improving local 
accountability. 

Furthermore, the process creates a vocabulary for examining the effects of state and 
federal programs on local systems. We saw ample evidence of the influence of the larger 
state and federal context on local accountability relationships. In some cases these pressures 
enhance the accoimtability system. In other cases they are at odds with the natural 
accountability relationships that exist at the local level. Our description of the local 
accoimtability systems serves as a basis for examining the effects of federal and state 
initiatives in this area. Exploring this topic is the subject of a subsequent report. 



